Note: This is test shard 4 of 8.
[==========] Running 7 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from LocationAwareRebalancingBasicTest
[ RUN ] LocationAwareRebalancingBasicTest.Basic
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240501 14:03:43.970582 30194 test_util.cc:268] Using random seed: -195429140
I20240501 14:03:43.982378 30194 ts_itest-base.cc:112] Starting cluster with:
I20240501 14:03:43.982456 30194 ts_itest-base.cc:113] --------------
I20240501 14:03:43.982501 30194 ts_itest-base.cc:114] 6 tablet servers
I20240501 14:03:43.982556 30194 ts_itest-base.cc:115] 3 replicas per TS
I20240501 14:03:43.982614 30194 ts_itest-base.cc:116] --------------
2024-05-01T14:03:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T14:03:43Z Disabled control of system clock
I20240501 14:03:44.003156 30194 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUv7aek/build/asan/bin/kudu
/tmp/dist-test-taskUv7aek/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.124.190:35979
--webserver_interface=127.29.124.190
--webserver_port=0
--builtin_ntp_servers=127.29.124.148:42681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.124.190:35979
--location_mapping_cmd=/tmp/dist-test-taskUv7aek/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/location-assignment.state --map /A:2 --map /B:2 --map /C:2
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240501 14:03:44.307250 30206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:03:44.307551 30206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:03:44.307641 30206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:03:44.314929 30206 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:03:44.315052 30206 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 14:03:44.315106 30206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:03:44.315160 30206 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 14:03:44.315203 30206 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 14:03:44.323868 30206 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.124.148:42681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--location_mapping_cmd=/tmp/dist-test-taskUv7aek/build/asan/bin/testdata/assign-location.py --state_store=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/location-assignment.state --map /A:2 --map /B:2 --map /C:2
--ipki_ca_key_size=768
--master_addresses=127.29.124.190:35979
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.124.190:35979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.29.124.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:03:44.325117 30206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:03:44.326795 30206 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:03:44.337039 30211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:03:44.337148 30212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:03:44.337428 30214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:03:44.337812 30206 server_base.cc:1034] running on GCE node
I20240501 14:03:44.338439 30206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:03:44.339684 30206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:03:44.340883 30206 hybrid_clock.cc:648] HybridClock initialized: now 1714572224340844 us; error 26 us; skew 500 ppm
I20240501 14:03:44.341225 30206 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:03:44.346952 30206 webserver.cc:460] Webserver started at http://127.29.124.190:36865/ using document root <none> and password file <none>
I20240501 14:03:44.347635 30206 fs_manager.cc:359] Metadata directory not provided
I20240501 14:03:44.347736 30206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:03:44.348017 30206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:03:44.350499 30206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "83cd42ea625644ddab1b8a28f3b2898d"
format_stamp: "Formatted at 2024-05-01 14:03:44 on dist-test-slave-p83k"
I20240501 14:03:44.351202 30206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "83cd42ea625644ddab1b8a28f3b2898d"
format_stamp: "Formatted at 2024-05-01 14:03:44 on dist-test-slave-p83k"
I20240501 14:03:44.355868 30206 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20240501 14:03:44.358858 30220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:03:44.360044 30206 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20240501 14:03:44.360188 30206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/wal
uuid: "83cd42ea625644ddab1b8a28f3b2898d"
format_stamp: "Formatted at 2024-05-01 14:03:44 on dist-test-slave-p83k"
I20240501 14:03:44.360376 30206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:03:44.378803 30206 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:03:44.379530 30206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:03:44.379762 30206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:03:44.395707 30206 rpc_server.cc:307] RPC server started. Bound to: 127.29.124.190:35979
I20240501 14:03:44.395713 30271 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.124.190:35979 every 8 connection(s)
I20240501 14:03:44.396973 30206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 14:03:44.400461 30272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:44.406726 30194 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUv7aek/build/asan/bin/kudu as pid 30206
I20240501 14:03:44.406952 30194 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 14:03:44.408921 30272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83cd42ea625644ddab1b8a28f3b2898d: Bootstrap starting.
I20240501 14:03:44.412374 30272 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 83cd42ea625644ddab1b8a28f3b2898d: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:44.413667 30272 log.cc:824] T 00000000000000000000000000000000 P 83cd42ea625644ddab1b8a28f3b2898d: Log is configured to *not* fsync() on all Append() calls
I20240501 14:03:44.415989 30272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 83cd42ea625644ddab1b8a28f3b2898d: No bootstrap required, opened a new log
I20240501 14:03:44.422439 30272 raft_consensus.cc:348] T 00000000000000000000000000000000 P 83cd42ea625644ddab1b8a28f3b2898d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83cd42ea625644ddab1b8a28f3b2898d" member_type: VOTER last_known_addr { host: "127.29.124.190" port: 35979 } }
I20240501 14:03:44.422770 30272 raft_consensus.cc:374] T 00000000000000000000000000000000 P 83cd42ea625644ddab1b8a28f3b2898d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:44.422885 30272 raft_consensus.cc:729] T 00000000000000000000000000000000 P 83cd42ea625644ddab1b8a28f3b2898d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83cd42ea625644ddab1b8a28f3b2898d, State: Initialized, Role: FOLLOWER
I20240501 14:03:44.423460 30272 consensus_queue.cc:260] T 00000000000000000000000000000000 P 83cd42ea625644ddab1b8a28f3b2898d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83cd42ea625644ddab1b8a28f3b2898d" member_type: VOTER last_known_addr { host: "127.29.124.190" port: 35979 } }
I20240501 14:03:44.423650 30272 raft_consensus.cc:388] T 00000000000000000000000000000000 P 83cd42ea625644ddab1b8a28f3b2898d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:03:44.423761 30272 raft_consensus.cc:482] T 00000000000000000000000000000000 P 83cd42ea625644ddab1b8a28f3b2898d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:03:44.423887 30272 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 83cd42ea625644ddab1b8a28f3b2898d [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:44.425961 30272 raft_consensus.cc:504] T 00000000000000000000000000000000 P 83cd42ea625644ddab1b8a28f3b2898d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83cd42ea625644ddab1b8a28f3b2898d" member_type: VOTER last_known_addr { host: "127.29.124.190" port: 35979 } }
I20240501 14:03:44.426385 30272 leader_election.cc:302] T 00000000000000000000000000000000 P 83cd42ea625644ddab1b8a28f3b2898d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 83cd42ea625644ddab1b8a28f3b2898d; no voters:
I20240501 14:03:44.427080 30272 leader_election.cc:288] T 00000000000000000000000000000000 P 83cd42ea625644ddab1b8a28f3b2898d [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:03:44.427125 30277 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 83cd42ea625644ddab1b8a28f3b2898d [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:03:44.428350 30277 raft_consensus.cc:686] T 00000000000000000000000000000000 P 83cd42ea625644ddab1b8a28f3b2898d [term 1 LEADER]: Becoming Leader. State: Replica: 83cd42ea625644ddab1b8a28f3b2898d, State: Running, Role: LEADER
I20240501 14:03:44.428848 30277 consensus_queue.cc:237] T 00000000000000000000000000000000 P 83cd42ea625644ddab1b8a28f3b2898d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83cd42ea625644ddab1b8a28f3b2898d" member_type: VOTER last_known_addr { host: "127.29.124.190" port: 35979 } }
I20240501 14:03:44.429841 30272 sys_catalog.cc:564] T 00000000000000000000000000000000 P 83cd42ea625644ddab1b8a28f3b2898d [sys.catalog]: configured and running, proceeding with master startup.
I20240501 14:03:44.432922 30279 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83cd42ea625644ddab1b8a28f3b2898d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 83cd42ea625644ddab1b8a28f3b2898d. Latest consensus state: current_term: 1 leader_uuid: "83cd42ea625644ddab1b8a28f3b2898d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83cd42ea625644ddab1b8a28f3b2898d" member_type: VOTER last_known_addr { host: "127.29.124.190" port: 35979 } } }
I20240501 14:03:44.433566 30279 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83cd42ea625644ddab1b8a28f3b2898d [sys.catalog]: This master's current role is: LEADER
I20240501 14:03:44.433420 30278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 83cd42ea625644ddab1b8a28f3b2898d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "83cd42ea625644ddab1b8a28f3b2898d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "83cd42ea625644ddab1b8a28f3b2898d" member_type: VOTER last_known_addr { host: "127.29.124.190" port: 35979 } } }
I20240501 14:03:44.433733 30278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 83cd42ea625644ddab1b8a28f3b2898d [sys.catalog]: This master's current role is: LEADER
I20240501 14:03:44.438026 30286 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 14:03:44.444434 30286 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 14:03:44.452495 30286 catalog_manager.cc:1347] Generated new cluster ID: 4292aa6789974297adccd50ed9af43df
I20240501 14:03:44.452586 30286 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 14:03:44.472030 30286 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 14:03:44.473210 30286 catalog_manager.cc:1504] Loading token signing keys...
I20240501 14:03:44.492801 30286 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 83cd42ea625644ddab1b8a28f3b2898d: Generated new TSK 0
I20240501 14:03:44.493744 30286 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 14:03:44.509006 30194 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUv7aek/build/asan/bin/kudu
/tmp/dist-test-taskUv7aek/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.124.129:0
--local_ip_for_outbound_sockets=127.29.124.129
--webserver_interface=127.29.124.129
--webserver_port=0
--tserver_master_addrs=127.29.124.190:35979
--builtin_ntp_servers=127.29.124.148:42681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 14:03:44.820307 30296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:03:44.820570 30296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:03:44.820663 30296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:03:44.827782 30296 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:03:44.827893 30296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:03:44.828012 30296 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.124.129
I20240501 14:03:44.836859 30296 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.124.148:42681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.124.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.29.124.129
--webserver_port=0
--tserver_master_addrs=127.29.124.190:35979
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.124.129
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:03:44.838202 30296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:03:44.839865 30296 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:03:44.850728 30301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:03:44.851078 30302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:03:44.851373 30304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:03:44.851608 30296 server_base.cc:1034] running on GCE node
I20240501 14:03:44.852118 30296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:03:44.852936 30296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:03:44.854123 30296 hybrid_clock.cc:648] HybridClock initialized: now 1714572224854077 us; error 35 us; skew 500 ppm
I20240501 14:03:44.854398 30296 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:03:44.856366 30296 webserver.cc:460] Webserver started at http://127.29.124.129:46089/ using document root <none> and password file <none>
I20240501 14:03:44.857040 30296 fs_manager.cc:359] Metadata directory not provided
I20240501 14:03:44.857133 30296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:03:44.857439 30296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:03:44.859829 30296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "c0fad54e965647aaaa34b56674646cb4"
format_stamp: "Formatted at 2024-05-01 14:03:44 on dist-test-slave-p83k"
I20240501 14:03:44.860491 30296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "c0fad54e965647aaaa34b56674646cb4"
format_stamp: "Formatted at 2024-05-01 14:03:44 on dist-test-slave-p83k"
I20240501 14:03:44.865131 30296 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20240501 14:03:44.868244 30310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:03:44.869522 30296 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:03:44.869668 30296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "c0fad54e965647aaaa34b56674646cb4"
format_stamp: "Formatted at 2024-05-01 14:03:44 on dist-test-slave-p83k"
I20240501 14:03:44.869849 30296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:03:44.890610 30296 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:03:44.891362 30296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:03:44.891608 30296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:03:44.892488 30296 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:03:44.893950 30296 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:03:44.894047 30296 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:03:44.894152 30296 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:03:44.894218 30296 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:03:44.921622 30296 rpc_server.cc:307] RPC server started. Bound to: 127.29.124.129:42363
I20240501 14:03:44.921669 30422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.124.129:42363 every 8 connection(s)
I20240501 14:03:44.922820 30296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 14:03:44.924677 30194 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUv7aek/build/asan/bin/kudu as pid 30296
I20240501 14:03:44.924860 30194 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 14:03:44.928478 30194 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUv7aek/build/asan/bin/kudu
/tmp/dist-test-taskUv7aek/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.124.130:0
--local_ip_for_outbound_sockets=127.29.124.130
--webserver_interface=127.29.124.130
--webserver_port=0
--tserver_master_addrs=127.29.124.190:35979
--builtin_ntp_servers=127.29.124.148:42681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:03:44.935969 30423 heartbeater.cc:347] Connected to a master server at 127.29.124.190:35979
I20240501 14:03:44.936236 30423 heartbeater.cc:464] Registering TS with master...
I20240501 14:03:44.936962 30423 heartbeater.cc:511] Master 127.29.124.190:35979 requested a full tablet report, sending...
I20240501 14:03:44.979856 30237 ts_manager.cc:194] Registered new tserver with Master: c0fad54e965647aaaa34b56674646cb4 (127.29.124.129:42363)
I20240501 14:03:44.982350 30237 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.124.129:59397
W20240501 14:03:45.237578 30426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:03:45.237836 30426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:03:45.237932 30426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:03:45.245000 30426 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:03:45.245105 30426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:03:45.245215 30426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.124.130
I20240501 14:03:45.253772 30426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.124.148:42681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.124.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.29.124.130
--webserver_port=0
--tserver_master_addrs=127.29.124.190:35979
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.124.130
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:03:45.254946 30426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:03:45.256596 30426 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:03:45.267510 30433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:03:45.267928 30435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:03:45.268224 30432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:03:45.268585 30426 server_base.cc:1034] running on GCE node
I20240501 14:03:45.269111 30426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:03:45.269987 30426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:03:45.271188 30426 hybrid_clock.cc:648] HybridClock initialized: now 1714572225271141 us; error 41 us; skew 500 ppm
I20240501 14:03:45.271476 30426 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:03:45.273523 30426 webserver.cc:460] Webserver started at http://127.29.124.130:44615/ using document root <none> and password file <none>
I20240501 14:03:45.274235 30426 fs_manager.cc:359] Metadata directory not provided
I20240501 14:03:45.274340 30426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:03:45.274641 30426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:03:45.277155 30426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "8883498098f4454bbfe563fb83d31899"
format_stamp: "Formatted at 2024-05-01 14:03:45 on dist-test-slave-p83k"
I20240501 14:03:45.277969 30426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "8883498098f4454bbfe563fb83d31899"
format_stamp: "Formatted at 2024-05-01 14:03:45 on dist-test-slave-p83k"
I20240501 14:03:45.282789 30426 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240501 14:03:45.285892 30441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:03:45.287110 30426 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 14:03:45.287253 30426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "8883498098f4454bbfe563fb83d31899"
format_stamp: "Formatted at 2024-05-01 14:03:45 on dist-test-slave-p83k"
I20240501 14:03:45.287431 30426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:03:45.318145 30426 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:03:45.318969 30426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:03:45.319248 30426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:03:45.320180 30426 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:03:45.321656 30426 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:03:45.321755 30426 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:03:45.321869 30426 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:03:45.321975 30426 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20240501 14:03:45.349905 30426 rpc_server.cc:307] RPC server started. Bound to: 127.29.124.130:43099
I20240501 14:03:45.349965 30553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.124.130:43099 every 8 connection(s)
I20240501 14:03:45.351148 30426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 14:03:45.353852 30194 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUv7aek/build/asan/bin/kudu as pid 30426
I20240501 14:03:45.354022 30194 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 14:03:45.357322 30194 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUv7aek/build/asan/bin/kudu
/tmp/dist-test-taskUv7aek/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.124.131:0
--local_ip_for_outbound_sockets=127.29.124.131
--webserver_interface=127.29.124.131
--webserver_port=0
--tserver_master_addrs=127.29.124.190:35979
--builtin_ntp_servers=127.29.124.148:42681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:03:45.363708 30554 heartbeater.cc:347] Connected to a master server at 127.29.124.190:35979
I20240501 14:03:45.363986 30554 heartbeater.cc:464] Registering TS with master...
I20240501 14:03:45.364761 30554 heartbeater.cc:511] Master 127.29.124.190:35979 requested a full tablet report, sending...
I20240501 14:03:45.403206 30237 ts_manager.cc:194] Registered new tserver with Master: 8883498098f4454bbfe563fb83d31899 (127.29.124.130:43099)
I20240501 14:03:45.404135 30237 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.124.130:54315
W20240501 14:03:45.667255 30557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:03:45.667527 30557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:03:45.667622 30557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:03:45.674975 30557 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:03:45.675096 30557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:03:45.675210 30557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.124.131
I20240501 14:03:45.683949 30557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.124.148:42681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.124.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.29.124.131
--webserver_port=0
--tserver_master_addrs=127.29.124.190:35979
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.124.131
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:03:45.685132 30557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:03:45.686786 30557 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:03:45.697896 30563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:03:45.698334 30566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:03:45.698657 30564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:03:45.699085 30557 server_base.cc:1034] running on GCE node
I20240501 14:03:45.699611 30557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:03:45.700433 30557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:03:45.701612 30557 hybrid_clock.cc:648] HybridClock initialized: now 1714572225701578 us; error 25 us; skew 500 ppm
I20240501 14:03:45.701879 30557 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:03:45.703838 30557 webserver.cc:460] Webserver started at http://127.29.124.131:40699/ using document root <none> and password file <none>
I20240501 14:03:45.704504 30557 fs_manager.cc:359] Metadata directory not provided
I20240501 14:03:45.704592 30557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:03:45.704840 30557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:03:45.707536 30557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "a3e62ccad691413ea2554182f73418a3"
format_stamp: "Formatted at 2024-05-01 14:03:45 on dist-test-slave-p83k"
I20240501 14:03:45.708227 30557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "a3e62ccad691413ea2554182f73418a3"
format_stamp: "Formatted at 2024-05-01 14:03:45 on dist-test-slave-p83k"
I20240501 14:03:45.713173 30557 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.006s
I20240501 14:03:45.716262 30572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:03:45.717530 30557 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20240501 14:03:45.717655 30557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "a3e62ccad691413ea2554182f73418a3"
format_stamp: "Formatted at 2024-05-01 14:03:45 on dist-test-slave-p83k"
I20240501 14:03:45.717813 30557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:03:45.737655 30557 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:03:45.738451 30557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:03:45.738680 30557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:03:45.739542 30557 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:03:45.741236 30557 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:03:45.741356 30557 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20240501 14:03:45.741451 30557 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:03:45.741513 30557 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:03:45.769812 30557 rpc_server.cc:307] RPC server started. Bound to: 127.29.124.131:43965
I20240501 14:03:45.769862 30684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.124.131:43965 every 8 connection(s)
I20240501 14:03:45.771092 30557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 14:03:45.781028 30194 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUv7aek/build/asan/bin/kudu as pid 30557
I20240501 14:03:45.781225 30194 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 14:03:45.783526 30685 heartbeater.cc:347] Connected to a master server at 127.29.124.190:35979
I20240501 14:03:45.783814 30685 heartbeater.cc:464] Registering TS with master...
I20240501 14:03:45.784490 30194 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUv7aek/build/asan/bin/kudu
/tmp/dist-test-taskUv7aek/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.124.132:0
--local_ip_for_outbound_sockets=127.29.124.132
--webserver_interface=127.29.124.132
--webserver_port=0
--tserver_master_addrs=127.29.124.190:35979
--builtin_ntp_servers=127.29.124.148:42681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:03:45.784574 30685 heartbeater.cc:511] Master 127.29.124.190:35979 requested a full tablet report, sending...
I20240501 14:03:45.822959 30237 ts_manager.cc:194] Registered new tserver with Master: a3e62ccad691413ea2554182f73418a3 (127.29.124.131:43965)
I20240501 14:03:45.823874 30237 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.124.131:52407
I20240501 14:03:45.985822 30423 heartbeater.cc:503] Master 127.29.124.190:35979 was elected leader, sending a full tablet report...
W20240501 14:03:46.094121 30688 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:03:46.094410 30688 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:03:46.094524 30688 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:03:46.101751 30688 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:03:46.101878 30688 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:03:46.102022 30688 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.124.132
I20240501 14:03:46.111326 30688 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.124.148:42681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.124.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.29.124.132
--webserver_port=0
--tserver_master_addrs=127.29.124.190:35979
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.124.132
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:03:46.112584 30688 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:03:46.114338 30688 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:03:46.125444 30694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:03:46.125897 30695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:03:46.126214 30697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:03:46.126600 30688 server_base.cc:1034] running on GCE node
I20240501 14:03:46.127138 30688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:03:46.127986 30688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:03:46.129186 30688 hybrid_clock.cc:648] HybridClock initialized: now 1714572226129134 us; error 43 us; skew 500 ppm
I20240501 14:03:46.129501 30688 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:03:46.131703 30688 webserver.cc:460] Webserver started at http://127.29.124.132:36129/ using document root <none> and password file <none>
I20240501 14:03:46.132401 30688 fs_manager.cc:359] Metadata directory not provided
I20240501 14:03:46.132509 30688 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:03:46.132798 30688 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:03:46.135478 30688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "870fbdcfdac84bacae0ad22dba25f626"
format_stamp: "Formatted at 2024-05-01 14:03:46 on dist-test-slave-p83k"
I20240501 14:03:46.136245 30688 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "870fbdcfdac84bacae0ad22dba25f626"
format_stamp: "Formatted at 2024-05-01 14:03:46 on dist-test-slave-p83k"
I20240501 14:03:46.141481 30688 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20240501 14:03:46.144744 30703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:03:46.146095 30688 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:03:46.146241 30688 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "870fbdcfdac84bacae0ad22dba25f626"
format_stamp: "Formatted at 2024-05-01 14:03:46 on dist-test-slave-p83k"
I20240501 14:03:46.146430 30688 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:03:46.170671 30688 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:03:46.171519 30688 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:03:46.171783 30688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:03:46.172750 30688 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:03:46.174320 30688 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:03:46.174407 30688 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:03:46.174511 30688 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:03:46.174572 30688 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:03:46.202720 30688 rpc_server.cc:307] RPC server started. Bound to: 127.29.124.132:44267
I20240501 14:03:46.202751 30815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.124.132:44267 every 8 connection(s)
I20240501 14:03:46.204012 30688 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 14:03:46.209575 30194 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUv7aek/build/asan/bin/kudu as pid 30688
I20240501 14:03:46.209749 30194 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 14:03:46.212621 30194 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUv7aek/build/asan/bin/kudu
/tmp/dist-test-taskUv7aek/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.124.133:0
--local_ip_for_outbound_sockets=127.29.124.133
--webserver_interface=127.29.124.133
--webserver_port=0
--tserver_master_addrs=127.29.124.190:35979
--builtin_ntp_servers=127.29.124.148:42681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:03:46.216663 30816 heartbeater.cc:347] Connected to a master server at 127.29.124.190:35979
I20240501 14:03:46.216940 30816 heartbeater.cc:464] Registering TS with master...
I20240501 14:03:46.217700 30816 heartbeater.cc:511] Master 127.29.124.190:35979 requested a full tablet report, sending...
I20240501 14:03:46.269690 30237 ts_manager.cc:194] Registered new tserver with Master: 870fbdcfdac84bacae0ad22dba25f626 (127.29.124.132:44267)
I20240501 14:03:46.270637 30237 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.124.132:52769
I20240501 14:03:46.406781 30554 heartbeater.cc:503] Master 127.29.124.190:35979 was elected leader, sending a full tablet report...
W20240501 14:03:46.529506 30819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:03:46.529768 30819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:03:46.529860 30819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:03:46.536947 30819 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:03:46.537066 30819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:03:46.537216 30819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.124.133
I20240501 14:03:46.545759 30819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.124.148:42681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.124.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.29.124.133
--webserver_port=0
--tserver_master_addrs=127.29.124.190:35979
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.124.133
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:03:46.546993 30819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:03:46.548627 30819 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:03:46.559650 30825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:03:46.560032 30826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:03:46.560338 30828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:03:46.560866 30819 server_base.cc:1034] running on GCE node
I20240501 14:03:46.561450 30819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:03:46.562294 30819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:03:46.563494 30819 hybrid_clock.cc:648] HybridClock initialized: now 1714572226563448 us; error 37 us; skew 500 ppm
I20240501 14:03:46.563786 30819 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:03:46.565922 30819 webserver.cc:460] Webserver started at http://127.29.124.133:40055/ using document root <none> and password file <none>
I20240501 14:03:46.566624 30819 fs_manager.cc:359] Metadata directory not provided
I20240501 14:03:46.566731 30819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:03:46.567018 30819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:03:46.569558 30819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "99f6ca2d65d94b33b7a1d2341c24179a"
format_stamp: "Formatted at 2024-05-01 14:03:46 on dist-test-slave-p83k"
I20240501 14:03:46.570250 30819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "99f6ca2d65d94b33b7a1d2341c24179a"
format_stamp: "Formatted at 2024-05-01 14:03:46 on dist-test-slave-p83k"
I20240501 14:03:46.575100 30819 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 14:03:46.578277 30834 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:03:46.579512 30819 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240501 14:03:46.579635 30819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "99f6ca2d65d94b33b7a1d2341c24179a"
format_stamp: "Formatted at 2024-05-01 14:03:46 on dist-test-slave-p83k"
I20240501 14:03:46.579792 30819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:03:46.605871 30819 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:03:46.606657 30819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:03:46.606885 30819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:03:46.607750 30819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:03:46.609105 30819 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:03:46.609185 30819 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:03:46.609282 30819 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:03:46.609359 30819 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20240501 14:03:46.637208 30819 rpc_server.cc:307] RPC server started. Bound to: 127.29.124.133:35195
I20240501 14:03:46.637327 30946 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.124.133:35195 every 8 connection(s)
I20240501 14:03:46.638542 30819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 14:03:46.646173 30194 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUv7aek/build/asan/bin/kudu as pid 30819
I20240501 14:03:46.646349 30194 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 14:03:46.649758 30194 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUv7aek/build/asan/bin/kudu
/tmp/dist-test-taskUv7aek/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.124.134:0
--local_ip_for_outbound_sockets=127.29.124.134
--webserver_interface=127.29.124.134
--webserver_port=0
--tserver_master_addrs=127.29.124.190:35979
--builtin_ntp_servers=127.29.124.148:42681
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:03:46.651000 30947 heartbeater.cc:347] Connected to a master server at 127.29.124.190:35979
I20240501 14:03:46.651326 30947 heartbeater.cc:464] Registering TS with master...
I20240501 14:03:46.652297 30947 heartbeater.cc:511] Master 127.29.124.190:35979 requested a full tablet report, sending...
I20240501 14:03:46.692281 30237 ts_manager.cc:194] Registered new tserver with Master: 99f6ca2d65d94b33b7a1d2341c24179a (127.29.124.133:35195)
I20240501 14:03:46.693733 30237 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.124.133:38841
I20240501 14:03:46.826483 30685 heartbeater.cc:503] Master 127.29.124.190:35979 was elected leader, sending a full tablet report...
W20240501 14:03:46.963578 30950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:03:46.963837 30950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:03:46.963929 30950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:03:46.971136 30950 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:03:46.971243 30950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:03:46.971359 30950 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.124.134
I20240501 14:03:46.980144 30950 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.124.148:42681
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.124.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.29.124.134
--webserver_port=0
--tserver_master_addrs=127.29.124.190:35979
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.124.134
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:03:46.981407 30950 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:03:46.983073 30950 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:03:46.995007 30956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:03:46.995224 30957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:03:46.995708 30959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:03:46.996064 30950 server_base.cc:1034] running on GCE node
I20240501 14:03:46.996613 30950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:03:46.997550 30950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:03:46.998749 30950 hybrid_clock.cc:648] HybridClock initialized: now 1714572226998701 us; error 39 us; skew 500 ppm
I20240501 14:03:46.999075 30950 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:03:47.001148 30950 webserver.cc:460] Webserver started at http://127.29.124.134:33885/ using document root <none> and password file <none>
I20240501 14:03:47.001899 30950 fs_manager.cc:359] Metadata directory not provided
I20240501 14:03:47.002019 30950 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:03:47.002315 30950 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:03:47.004869 30950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "9964c1c76dd14f7dbce766b0c022ab51"
format_stamp: "Formatted at 2024-05-01 14:03:46 on dist-test-slave-p83k"
I20240501 14:03:47.005647 30950 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "9964c1c76dd14f7dbce766b0c022ab51"
format_stamp: "Formatted at 2024-05-01 14:03:46 on dist-test-slave-p83k"
I20240501 14:03:47.010862 30950 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240501 14:03:47.014400 30965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:03:47.015877 30950 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240501 14:03:47.016067 30950 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "9964c1c76dd14f7dbce766b0c022ab51"
format_stamp: "Formatted at 2024-05-01 14:03:46 on dist-test-slave-p83k"
I20240501 14:03:47.016269 30950 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:03:47.033074 30950 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:03:47.033924 30950 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:03:47.034183 30950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:03:47.035053 30950 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:03:47.036422 30950 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:03:47.036511 30950 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:03:47.036617 30950 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:03:47.036682 30950 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:03:47.065354 30950 rpc_server.cc:307] RPC server started. Bound to: 127.29.124.134:38101
I20240501 14:03:47.065421 31077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.124.134:38101 every 8 connection(s)
I20240501 14:03:47.066642 30950 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 14:03:47.070446 30194 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUv7aek/build/asan/bin/kudu as pid 30950
I20240501 14:03:47.070626 30194 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 14:03:47.079282 31078 heartbeater.cc:347] Connected to a master server at 127.29.124.190:35979
I20240501 14:03:47.079551 31078 heartbeater.cc:464] Registering TS with master...
I20240501 14:03:47.080294 31078 heartbeater.cc:511] Master 127.29.124.190:35979 requested a full tablet report, sending...
I20240501 14:03:47.119148 30237 ts_manager.cc:194] Registered new tserver with Master: 9964c1c76dd14f7dbce766b0c022ab51 (127.29.124.134:38101)
I20240501 14:03:47.120061 30237 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.124.134:55467
I20240501 14:03:47.124855 30194 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240501 14:03:47.143980 30194 external_mini_cluster.cc:930] 6 TS(s) registered with all masters
I20240501 14:03:47.273365 30816 heartbeater.cc:503] Master 127.29.124.190:35979 was elected leader, sending a full tablet report...
I20240501 14:03:47.696409 30947 heartbeater.cc:503] Master 127.29.124.190:35979 was elected leader, sending a full tablet report...
I20240501 14:03:48.122957 31078 heartbeater.cc:503] Master 127.29.124.190:35979 was elected leader, sending a full tablet report...
I20240501 14:03:50.897696 30237 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35960:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:03:50.956353 30881 tablet_service.cc:1467] Processing CreateTablet for tablet 55f4202b341f42eb8e132766b634263b (DEFAULT_TABLE table=rebalance_test_table_0 [id=7c80b0d5116f4ab2a6be322a675a6e3d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:50.957408 30882 tablet_service.cc:1467] Processing CreateTablet for tablet cd60b6c4ba4b4eb58326755b08efb0e9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7c80b0d5116f4ab2a6be322a675a6e3d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:50.958472 30882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd60b6c4ba4b4eb58326755b08efb0e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:50.959164 30881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55f4202b341f42eb8e132766b634263b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:50.959538 30751 tablet_service.cc:1467] Processing CreateTablet for tablet cd60b6c4ba4b4eb58326755b08efb0e9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7c80b0d5116f4ab2a6be322a675a6e3d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:50.961787 30357 tablet_service.cc:1467] Processing CreateTablet for tablet 55f4202b341f42eb8e132766b634263b (DEFAULT_TABLE table=rebalance_test_table_0 [id=7c80b0d5116f4ab2a6be322a675a6e3d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:50.963001 30357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55f4202b341f42eb8e132766b634263b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:50.963186 30489 tablet_service.cc:1467] Processing CreateTablet for tablet cd60b6c4ba4b4eb58326755b08efb0e9 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7c80b0d5116f4ab2a6be322a675a6e3d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:50.964509 30489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd60b6c4ba4b4eb58326755b08efb0e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:50.964594 30619 tablet_service.cc:1467] Processing CreateTablet for tablet 55f4202b341f42eb8e132766b634263b (DEFAULT_TABLE table=rebalance_test_table_0 [id=7c80b0d5116f4ab2a6be322a675a6e3d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:50.966588 30619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55f4202b341f42eb8e132766b634263b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:50.969163 30620 tablet_service.cc:1467] Processing CreateTablet for tablet c6f8a2b1f923426e989e4d2dbcac9e2d (DEFAULT_TABLE table=rebalance_test_table_0 [id=7c80b0d5116f4ab2a6be322a675a6e3d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:50.969666 30620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6f8a2b1f923426e989e4d2dbcac9e2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:50.970023 30751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cd60b6c4ba4b4eb58326755b08efb0e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:50.956161 30358 tablet_service.cc:1467] Processing CreateTablet for tablet c6f8a2b1f923426e989e4d2dbcac9e2d (DEFAULT_TABLE table=rebalance_test_table_0 [id=7c80b0d5116f4ab2a6be322a675a6e3d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:50.973814 30358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6f8a2b1f923426e989e4d2dbcac9e2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:50.977242 31013 tablet_service.cc:1467] Processing CreateTablet for tablet c6f8a2b1f923426e989e4d2dbcac9e2d (DEFAULT_TABLE table=rebalance_test_table_0 [id=7c80b0d5116f4ab2a6be322a675a6e3d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:50.979142 31013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c6f8a2b1f923426e989e4d2dbcac9e2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:51.014050 31106 tablet_bootstrap.cc:492] T c6f8a2b1f923426e989e4d2dbcac9e2d P a3e62ccad691413ea2554182f73418a3: Bootstrap starting.
I20240501 14:03:51.023389 31106 tablet_bootstrap.cc:654] T c6f8a2b1f923426e989e4d2dbcac9e2d P a3e62ccad691413ea2554182f73418a3: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:51.024932 31106 log.cc:824] T c6f8a2b1f923426e989e4d2dbcac9e2d P a3e62ccad691413ea2554182f73418a3: Log is configured to *not* fsync() on all Append() calls
I20240501 14:03:51.027660 31103 tablet_bootstrap.cc:492] T 55f4202b341f42eb8e132766b634263b P 99f6ca2d65d94b33b7a1d2341c24179a: Bootstrap starting.
I20240501 14:03:51.028453 31105 tablet_bootstrap.cc:492] T cd60b6c4ba4b4eb58326755b08efb0e9 P 870fbdcfdac84bacae0ad22dba25f626: Bootstrap starting.
I20240501 14:03:51.029994 31104 tablet_bootstrap.cc:492] T c6f8a2b1f923426e989e4d2dbcac9e2d P c0fad54e965647aaaa34b56674646cb4: Bootstrap starting.
I20240501 14:03:51.030987 31107 tablet_bootstrap.cc:492] T cd60b6c4ba4b4eb58326755b08efb0e9 P 8883498098f4454bbfe563fb83d31899: Bootstrap starting.
I20240501 14:03:51.031608 31103 tablet_bootstrap.cc:654] T 55f4202b341f42eb8e132766b634263b P 99f6ca2d65d94b33b7a1d2341c24179a: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:51.032838 31103 log.cc:824] T 55f4202b341f42eb8e132766b634263b P 99f6ca2d65d94b33b7a1d2341c24179a: Log is configured to *not* fsync() on all Append() calls
I20240501 14:03:51.033309 31104 tablet_bootstrap.cc:654] T c6f8a2b1f923426e989e4d2dbcac9e2d P c0fad54e965647aaaa34b56674646cb4: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:51.034576 31104 log.cc:824] T c6f8a2b1f923426e989e4d2dbcac9e2d P c0fad54e965647aaaa34b56674646cb4: Log is configured to *not* fsync() on all Append() calls
I20240501 14:03:51.035207 31106 tablet_bootstrap.cc:492] T c6f8a2b1f923426e989e4d2dbcac9e2d P a3e62ccad691413ea2554182f73418a3: No bootstrap required, opened a new log
I20240501 14:03:51.035514 31106 ts_tablet_manager.cc:1367] T c6f8a2b1f923426e989e4d2dbcac9e2d P a3e62ccad691413ea2554182f73418a3: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.000s
I20240501 14:03:51.037724 31105 tablet_bootstrap.cc:654] T cd60b6c4ba4b4eb58326755b08efb0e9 P 870fbdcfdac84bacae0ad22dba25f626: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:51.038610 31107 tablet_bootstrap.cc:654] T cd60b6c4ba4b4eb58326755b08efb0e9 P 8883498098f4454bbfe563fb83d31899: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:51.039090 31105 log.cc:824] T cd60b6c4ba4b4eb58326755b08efb0e9 P 870fbdcfdac84bacae0ad22dba25f626: Log is configured to *not* fsync() on all Append() calls
I20240501 14:03:51.039798 31107 log.cc:824] T cd60b6c4ba4b4eb58326755b08efb0e9 P 8883498098f4454bbfe563fb83d31899: Log is configured to *not* fsync() on all Append() calls
I20240501 14:03:51.043504 31104 tablet_bootstrap.cc:492] T c6f8a2b1f923426e989e4d2dbcac9e2d P c0fad54e965647aaaa34b56674646cb4: No bootstrap required, opened a new log
I20240501 14:03:51.044862 31104 ts_tablet_manager.cc:1367] T c6f8a2b1f923426e989e4d2dbcac9e2d P c0fad54e965647aaaa34b56674646cb4: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.000s
I20240501 14:03:51.046684 31103 tablet_bootstrap.cc:492] T 55f4202b341f42eb8e132766b634263b P 99f6ca2d65d94b33b7a1d2341c24179a: No bootstrap required, opened a new log
I20240501 14:03:51.046980 31103 ts_tablet_manager.cc:1367] T 55f4202b341f42eb8e132766b634263b P 99f6ca2d65d94b33b7a1d2341c24179a: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.007s
I20240501 14:03:51.048038 31105 tablet_bootstrap.cc:492] T cd60b6c4ba4b4eb58326755b08efb0e9 P 870fbdcfdac84bacae0ad22dba25f626: No bootstrap required, opened a new log
I20240501 14:03:51.047909 31106 raft_consensus.cc:348] T c6f8a2b1f923426e989e4d2dbcac9e2d P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:03:51.048401 31106 raft_consensus.cc:374] T c6f8a2b1f923426e989e4d2dbcac9e2d P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:51.048445 31105 ts_tablet_manager.cc:1367] T cd60b6c4ba4b4eb58326755b08efb0e9 P 870fbdcfdac84bacae0ad22dba25f626: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.003s
I20240501 14:03:51.048507 31106 raft_consensus.cc:729] T c6f8a2b1f923426e989e4d2dbcac9e2d P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3e62ccad691413ea2554182f73418a3, State: Initialized, Role: FOLLOWER
I20240501 14:03:51.049925 31109 tablet_bootstrap.cc:492] T c6f8a2b1f923426e989e4d2dbcac9e2d P 9964c1c76dd14f7dbce766b0c022ab51: Bootstrap starting.
I20240501 14:03:51.050945 31106 consensus_queue.cc:260] T c6f8a2b1f923426e989e4d2dbcac9e2d P a3e62ccad691413ea2554182f73418a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:03:51.052866 31107 tablet_bootstrap.cc:492] T cd60b6c4ba4b4eb58326755b08efb0e9 P 8883498098f4454bbfe563fb83d31899: No bootstrap required, opened a new log
I20240501 14:03:51.053176 31107 ts_tablet_manager.cc:1367] T cd60b6c4ba4b4eb58326755b08efb0e9 P 8883498098f4454bbfe563fb83d31899: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.004s
I20240501 14:03:51.053642 31109 tablet_bootstrap.cc:654] T c6f8a2b1f923426e989e4d2dbcac9e2d P 9964c1c76dd14f7dbce766b0c022ab51: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:51.054986 31109 log.cc:824] T c6f8a2b1f923426e989e4d2dbcac9e2d P 9964c1c76dd14f7dbce766b0c022ab51: Log is configured to *not* fsync() on all Append() calls
I20240501 14:03:51.063506 31103 raft_consensus.cc:348] T 55f4202b341f42eb8e132766b634263b P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } }
I20240501 14:03:51.064023 31103 raft_consensus.cc:374] T 55f4202b341f42eb8e132766b634263b P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:51.064172 31103 raft_consensus.cc:729] T 55f4202b341f42eb8e132766b634263b P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99f6ca2d65d94b33b7a1d2341c24179a, State: Initialized, Role: FOLLOWER
I20240501 14:03:51.065438 31104 raft_consensus.cc:348] T c6f8a2b1f923426e989e4d2dbcac9e2d P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:03:51.065111 31103 consensus_queue.cc:260] T 55f4202b341f42eb8e132766b634263b P 99f6ca2d65d94b33b7a1d2341c24179a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } }
I20240501 14:03:51.066381 31104 raft_consensus.cc:374] T c6f8a2b1f923426e989e4d2dbcac9e2d P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:51.066522 31104 raft_consensus.cc:729] T c6f8a2b1f923426e989e4d2dbcac9e2d P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0fad54e965647aaaa34b56674646cb4, State: Initialized, Role: FOLLOWER
I20240501 14:03:51.067291 31104 consensus_queue.cc:260] T c6f8a2b1f923426e989e4d2dbcac9e2d P c0fad54e965647aaaa34b56674646cb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:03:51.067546 31105 raft_consensus.cc:348] T cd60b6c4ba4b4eb58326755b08efb0e9 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:03:51.068212 31107 raft_consensus.cc:348] T cd60b6c4ba4b4eb58326755b08efb0e9 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:03:51.068652 31107 raft_consensus.cc:374] T cd60b6c4ba4b4eb58326755b08efb0e9 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:51.068786 31107 raft_consensus.cc:729] T cd60b6c4ba4b4eb58326755b08efb0e9 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8883498098f4454bbfe563fb83d31899, State: Initialized, Role: FOLLOWER
I20240501 14:03:51.069545 31107 consensus_queue.cc:260] T cd60b6c4ba4b4eb58326755b08efb0e9 P 8883498098f4454bbfe563fb83d31899 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:03:51.070091 31105 raft_consensus.cc:374] T cd60b6c4ba4b4eb58326755b08efb0e9 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:51.070238 31105 raft_consensus.cc:729] T cd60b6c4ba4b4eb58326755b08efb0e9 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 870fbdcfdac84bacae0ad22dba25f626, State: Initialized, Role: FOLLOWER
I20240501 14:03:51.070508 31109 tablet_bootstrap.cc:492] T c6f8a2b1f923426e989e4d2dbcac9e2d P 9964c1c76dd14f7dbce766b0c022ab51: No bootstrap required, opened a new log
I20240501 14:03:51.070878 31109 ts_tablet_manager.cc:1367] T c6f8a2b1f923426e989e4d2dbcac9e2d P 9964c1c76dd14f7dbce766b0c022ab51: Time spent bootstrapping tablet: real 0.021s user 0.015s sys 0.005s
I20240501 14:03:51.071090 31105 consensus_queue.cc:260] T cd60b6c4ba4b4eb58326755b08efb0e9 P 870fbdcfdac84bacae0ad22dba25f626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:03:51.074777 31106 ts_tablet_manager.cc:1398] T c6f8a2b1f923426e989e4d2dbcac9e2d P a3e62ccad691413ea2554182f73418a3: Time spent starting tablet: real 0.039s user 0.014s sys 0.017s
I20240501 14:03:51.076189 31105 ts_tablet_manager.cc:1398] T cd60b6c4ba4b4eb58326755b08efb0e9 P 870fbdcfdac84bacae0ad22dba25f626: Time spent starting tablet: real 0.028s user 0.005s sys 0.006s
I20240501 14:03:51.076174 31104 ts_tablet_manager.cc:1398] T c6f8a2b1f923426e989e4d2dbcac9e2d P c0fad54e965647aaaa34b56674646cb4: Time spent starting tablet: real 0.027s user 0.012s sys 0.000s
I20240501 14:03:51.076602 31104 tablet_bootstrap.cc:492] T 55f4202b341f42eb8e132766b634263b P c0fad54e965647aaaa34b56674646cb4: Bootstrap starting.
I20240501 14:03:51.078245 31104 tablet_bootstrap.cc:654] T 55f4202b341f42eb8e132766b634263b P c0fad54e965647aaaa34b56674646cb4: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:51.079509 31106 tablet_bootstrap.cc:492] T 55f4202b341f42eb8e132766b634263b P a3e62ccad691413ea2554182f73418a3: Bootstrap starting.
I20240501 14:03:51.081154 31106 tablet_bootstrap.cc:654] T 55f4202b341f42eb8e132766b634263b P a3e62ccad691413ea2554182f73418a3: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:51.082509 31107 ts_tablet_manager.cc:1398] T cd60b6c4ba4b4eb58326755b08efb0e9 P 8883498098f4454bbfe563fb83d31899: Time spent starting tablet: real 0.029s user 0.012s sys 0.000s
I20240501 14:03:51.083264 31103 ts_tablet_manager.cc:1398] T 55f4202b341f42eb8e132766b634263b P 99f6ca2d65d94b33b7a1d2341c24179a: Time spent starting tablet: real 0.036s user 0.013s sys 0.001s
I20240501 14:03:51.083693 31103 tablet_bootstrap.cc:492] T cd60b6c4ba4b4eb58326755b08efb0e9 P 99f6ca2d65d94b33b7a1d2341c24179a: Bootstrap starting.
I20240501 14:03:51.085434 31103 tablet_bootstrap.cc:654] T cd60b6c4ba4b4eb58326755b08efb0e9 P 99f6ca2d65d94b33b7a1d2341c24179a: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:51.085700 31109 raft_consensus.cc:348] T c6f8a2b1f923426e989e4d2dbcac9e2d P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:03:51.086158 31109 raft_consensus.cc:374] T c6f8a2b1f923426e989e4d2dbcac9e2d P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:51.086280 31109 raft_consensus.cc:729] T c6f8a2b1f923426e989e4d2dbcac9e2d P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9964c1c76dd14f7dbce766b0c022ab51, State: Initialized, Role: FOLLOWER
I20240501 14:03:51.086799 31104 tablet_bootstrap.cc:492] T 55f4202b341f42eb8e132766b634263b P c0fad54e965647aaaa34b56674646cb4: No bootstrap required, opened a new log
I20240501 14:03:51.086997 31104 ts_tablet_manager.cc:1367] T 55f4202b341f42eb8e132766b634263b P c0fad54e965647aaaa34b56674646cb4: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20240501 14:03:51.087736 31109 consensus_queue.cc:260] T c6f8a2b1f923426e989e4d2dbcac9e2d P 9964c1c76dd14f7dbce766b0c022ab51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:03:51.088076 31104 raft_consensus.cc:348] T 55f4202b341f42eb8e132766b634263b P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } }
I20240501 14:03:51.088719 31104 raft_consensus.cc:374] T 55f4202b341f42eb8e132766b634263b P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:51.088850 31104 raft_consensus.cc:729] T 55f4202b341f42eb8e132766b634263b P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0fad54e965647aaaa34b56674646cb4, State: Initialized, Role: FOLLOWER
I20240501 14:03:51.089112 31104 consensus_queue.cc:260] T 55f4202b341f42eb8e132766b634263b P c0fad54e965647aaaa34b56674646cb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } }
I20240501 14:03:51.093765 31103 tablet_bootstrap.cc:492] T cd60b6c4ba4b4eb58326755b08efb0e9 P 99f6ca2d65d94b33b7a1d2341c24179a: No bootstrap required, opened a new log
I20240501 14:03:51.093973 31103 ts_tablet_manager.cc:1367] T cd60b6c4ba4b4eb58326755b08efb0e9 P 99f6ca2d65d94b33b7a1d2341c24179a: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20240501 14:03:51.095023 31103 raft_consensus.cc:348] T cd60b6c4ba4b4eb58326755b08efb0e9 P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:03:51.095290 31103 raft_consensus.cc:374] T cd60b6c4ba4b4eb58326755b08efb0e9 P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:51.095431 31103 raft_consensus.cc:729] T cd60b6c4ba4b4eb58326755b08efb0e9 P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99f6ca2d65d94b33b7a1d2341c24179a, State: Initialized, Role: FOLLOWER
I20240501 14:03:51.095698 31103 consensus_queue.cc:260] T cd60b6c4ba4b4eb58326755b08efb0e9 P 99f6ca2d65d94b33b7a1d2341c24179a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:03:51.096403 31103 ts_tablet_manager.cc:1398] T cd60b6c4ba4b4eb58326755b08efb0e9 P 99f6ca2d65d94b33b7a1d2341c24179a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:03:51.097051 31104 ts_tablet_manager.cc:1398] T 55f4202b341f42eb8e132766b634263b P c0fad54e965647aaaa34b56674646cb4: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20240501 14:03:51.100440 31106 tablet_bootstrap.cc:492] T 55f4202b341f42eb8e132766b634263b P a3e62ccad691413ea2554182f73418a3: No bootstrap required, opened a new log
I20240501 14:03:51.100600 31106 ts_tablet_manager.cc:1367] T 55f4202b341f42eb8e132766b634263b P a3e62ccad691413ea2554182f73418a3: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.005s
I20240501 14:03:51.100929 31109 ts_tablet_manager.cc:1398] T c6f8a2b1f923426e989e4d2dbcac9e2d P 9964c1c76dd14f7dbce766b0c022ab51: Time spent starting tablet: real 0.030s user 0.007s sys 0.009s
I20240501 14:03:51.101668 31106 raft_consensus.cc:348] T 55f4202b341f42eb8e132766b634263b P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } }
I20240501 14:03:51.101872 31106 raft_consensus.cc:374] T 55f4202b341f42eb8e132766b634263b P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:51.101967 31106 raft_consensus.cc:729] T 55f4202b341f42eb8e132766b634263b P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3e62ccad691413ea2554182f73418a3, State: Initialized, Role: FOLLOWER
I20240501 14:03:51.102177 31106 consensus_queue.cc:260] T 55f4202b341f42eb8e132766b634263b P a3e62ccad691413ea2554182f73418a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } }
I20240501 14:03:51.102618 31106 ts_tablet_manager.cc:1398] T 55f4202b341f42eb8e132766b634263b P a3e62ccad691413ea2554182f73418a3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20240501 14:03:51.106117 30555 tablet.cc:2369] T cd60b6c4ba4b4eb58326755b08efb0e9 P 8883498098f4454bbfe563fb83d31899: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 14:03:51.142448 30948 tablet.cc:2369] T 55f4202b341f42eb8e132766b634263b P 99f6ca2d65d94b33b7a1d2341c24179a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 14:03:51.177953 30424 tablet.cc:2369] T 55f4202b341f42eb8e132766b634263b P c0fad54e965647aaaa34b56674646cb4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:03:51.206465 31116 raft_consensus.cc:482] T 55f4202b341f42eb8e132766b634263b P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:03:51.206707 31116 raft_consensus.cc:504] T 55f4202b341f42eb8e132766b634263b P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } }
I20240501 14:03:51.208357 31116 leader_election.cc:288] T 55f4202b341f42eb8e132766b634263b P 99f6ca2d65d94b33b7a1d2341c24179a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0fad54e965647aaaa34b56674646cb4 (127.29.124.129:42363), a3e62ccad691413ea2554182f73418a3 (127.29.124.131:43965)
W20240501 14:03:51.210584 30817 tablet.cc:2369] T cd60b6c4ba4b4eb58326755b08efb0e9 P 870fbdcfdac84bacae0ad22dba25f626: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:03:51.211508 31114 raft_consensus.cc:482] T c6f8a2b1f923426e989e4d2dbcac9e2d P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:03:51.211728 31114 raft_consensus.cc:504] T c6f8a2b1f923426e989e4d2dbcac9e2d P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:03:51.213681 31114 leader_election.cc:288] T c6f8a2b1f923426e989e4d2dbcac9e2d P a3e62ccad691413ea2554182f73418a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0fad54e965647aaaa34b56674646cb4 (127.29.124.129:42363), 9964c1c76dd14f7dbce766b0c022ab51 (127.29.124.134:38101)
I20240501 14:03:51.217418 30378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55f4202b341f42eb8e132766b634263b" candidate_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0fad54e965647aaaa34b56674646cb4" is_pre_election: true
I20240501 14:03:51.217391 30640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55f4202b341f42eb8e132766b634263b" candidate_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3e62ccad691413ea2554182f73418a3" is_pre_election: true
I20240501 14:03:51.217926 30378 raft_consensus.cc:2454] T 55f4202b341f42eb8e132766b634263b P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99f6ca2d65d94b33b7a1d2341c24179a in term 0.
I20240501 14:03:51.217892 30640 raft_consensus.cc:2454] T 55f4202b341f42eb8e132766b634263b P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99f6ca2d65d94b33b7a1d2341c24179a in term 0.
I20240501 14:03:51.218638 30835 leader_election.cc:302] T 55f4202b341f42eb8e132766b634263b P 99f6ca2d65d94b33b7a1d2341c24179a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99f6ca2d65d94b33b7a1d2341c24179a, a3e62ccad691413ea2554182f73418a3; no voters:
I20240501 14:03:51.219283 31116 raft_consensus.cc:2781] T 55f4202b341f42eb8e132766b634263b P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:03:51.219431 31116 raft_consensus.cc:482] T 55f4202b341f42eb8e132766b634263b P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:03:51.219522 31116 raft_consensus.cc:3037] T 55f4202b341f42eb8e132766b634263b P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:51.221719 31116 raft_consensus.cc:504] T 55f4202b341f42eb8e132766b634263b P 99f6ca2d65d94b33b7a1d2341c24179a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } }
I20240501 14:03:51.222700 30378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55f4202b341f42eb8e132766b634263b" candidate_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0fad54e965647aaaa34b56674646cb4"
I20240501 14:03:51.222735 31116 leader_election.cc:288] T 55f4202b341f42eb8e132766b634263b P 99f6ca2d65d94b33b7a1d2341c24179a [CANDIDATE]: Term 1 election: Requested vote from peers c0fad54e965647aaaa34b56674646cb4 (127.29.124.129:42363), a3e62ccad691413ea2554182f73418a3 (127.29.124.131:43965)
I20240501 14:03:51.222882 30378 raft_consensus.cc:3037] T 55f4202b341f42eb8e132766b634263b P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:51.223229 30640 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55f4202b341f42eb8e132766b634263b" candidate_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3e62ccad691413ea2554182f73418a3"
I20240501 14:03:51.223397 30640 raft_consensus.cc:3037] T 55f4202b341f42eb8e132766b634263b P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:51.224025 31033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6f8a2b1f923426e989e4d2dbcac9e2d" candidate_uuid: "a3e62ccad691413ea2554182f73418a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9964c1c76dd14f7dbce766b0c022ab51" is_pre_election: true
I20240501 14:03:51.224390 31033 raft_consensus.cc:2454] T c6f8a2b1f923426e989e4d2dbcac9e2d P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3e62ccad691413ea2554182f73418a3 in term 0.
I20240501 14:03:51.225567 30573 leader_election.cc:302] T c6f8a2b1f923426e989e4d2dbcac9e2d P a3e62ccad691413ea2554182f73418a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9964c1c76dd14f7dbce766b0c022ab51, a3e62ccad691413ea2554182f73418a3; no voters:
I20240501 14:03:51.225766 30377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6f8a2b1f923426e989e4d2dbcac9e2d" candidate_uuid: "a3e62ccad691413ea2554182f73418a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0fad54e965647aaaa34b56674646cb4" is_pre_election: true
I20240501 14:03:51.225894 31114 raft_consensus.cc:2781] T c6f8a2b1f923426e989e4d2dbcac9e2d P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:03:51.226003 31114 raft_consensus.cc:482] T c6f8a2b1f923426e989e4d2dbcac9e2d P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:03:51.226011 30377 raft_consensus.cc:2454] T c6f8a2b1f923426e989e4d2dbcac9e2d P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3e62ccad691413ea2554182f73418a3 in term 0.
I20240501 14:03:51.226106 31114 raft_consensus.cc:3037] T c6f8a2b1f923426e989e4d2dbcac9e2d P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:51.226235 30640 raft_consensus.cc:2454] T 55f4202b341f42eb8e132766b634263b P a3e62ccad691413ea2554182f73418a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99f6ca2d65d94b33b7a1d2341c24179a in term 1.
I20240501 14:03:51.226461 30378 raft_consensus.cc:2454] T 55f4202b341f42eb8e132766b634263b P c0fad54e965647aaaa34b56674646cb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99f6ca2d65d94b33b7a1d2341c24179a in term 1.
I20240501 14:03:51.226621 30835 leader_election.cc:302] T 55f4202b341f42eb8e132766b634263b P 99f6ca2d65d94b33b7a1d2341c24179a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99f6ca2d65d94b33b7a1d2341c24179a, a3e62ccad691413ea2554182f73418a3; no voters:
I20240501 14:03:51.226878 31116 raft_consensus.cc:2781] T 55f4202b341f42eb8e132766b634263b P 99f6ca2d65d94b33b7a1d2341c24179a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:03:51.227248 31116 raft_consensus.cc:686] T 55f4202b341f42eb8e132766b634263b P 99f6ca2d65d94b33b7a1d2341c24179a [term 1 LEADER]: Becoming Leader. State: Replica: 99f6ca2d65d94b33b7a1d2341c24179a, State: Running, Role: LEADER
I20240501 14:03:51.227720 31116 consensus_queue.cc:237] T 55f4202b341f42eb8e132766b634263b P 99f6ca2d65d94b33b7a1d2341c24179a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } }
I20240501 14:03:51.228421 31114 raft_consensus.cc:504] T c6f8a2b1f923426e989e4d2dbcac9e2d P a3e62ccad691413ea2554182f73418a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:03:51.228945 31114 leader_election.cc:288] T c6f8a2b1f923426e989e4d2dbcac9e2d P a3e62ccad691413ea2554182f73418a3 [CANDIDATE]: Term 1 election: Requested vote from peers c0fad54e965647aaaa34b56674646cb4 (127.29.124.129:42363), 9964c1c76dd14f7dbce766b0c022ab51 (127.29.124.134:38101)
I20240501 14:03:51.229315 30378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6f8a2b1f923426e989e4d2dbcac9e2d" candidate_uuid: "a3e62ccad691413ea2554182f73418a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0fad54e965647aaaa34b56674646cb4"
I20240501 14:03:51.229476 30378 raft_consensus.cc:3037] T c6f8a2b1f923426e989e4d2dbcac9e2d P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:51.229440 31033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c6f8a2b1f923426e989e4d2dbcac9e2d" candidate_uuid: "a3e62ccad691413ea2554182f73418a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9964c1c76dd14f7dbce766b0c022ab51"
I20240501 14:03:51.229615 31033 raft_consensus.cc:3037] T c6f8a2b1f923426e989e4d2dbcac9e2d P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:51.231252 31119 raft_consensus.cc:482] T cd60b6c4ba4b4eb58326755b08efb0e9 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:03:51.231536 30378 raft_consensus.cc:2454] T c6f8a2b1f923426e989e4d2dbcac9e2d P c0fad54e965647aaaa34b56674646cb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3e62ccad691413ea2554182f73418a3 in term 1.
I20240501 14:03:51.231506 31119 raft_consensus.cc:504] T cd60b6c4ba4b4eb58326755b08efb0e9 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:03:51.231895 30574 leader_election.cc:302] T c6f8a2b1f923426e989e4d2dbcac9e2d P a3e62ccad691413ea2554182f73418a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3e62ccad691413ea2554182f73418a3, c0fad54e965647aaaa34b56674646cb4; no voters:
I20240501 14:03:51.232187 31114 raft_consensus.cc:2781] T c6f8a2b1f923426e989e4d2dbcac9e2d P a3e62ccad691413ea2554182f73418a3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:03:51.233717 31119 leader_election.cc:288] T cd60b6c4ba4b4eb58326755b08efb0e9 P 870fbdcfdac84bacae0ad22dba25f626 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99f6ca2d65d94b33b7a1d2341c24179a (127.29.124.133:35195), 8883498098f4454bbfe563fb83d31899 (127.29.124.130:43099)
I20240501 14:03:51.234251 31033 raft_consensus.cc:2454] T c6f8a2b1f923426e989e4d2dbcac9e2d P 9964c1c76dd14f7dbce766b0c022ab51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3e62ccad691413ea2554182f73418a3 in term 1.
I20240501 14:03:51.235070 31114 raft_consensus.cc:686] T c6f8a2b1f923426e989e4d2dbcac9e2d P a3e62ccad691413ea2554182f73418a3 [term 1 LEADER]: Becoming Leader. State: Replica: a3e62ccad691413ea2554182f73418a3, State: Running, Role: LEADER
I20240501 14:03:51.235597 31114 consensus_queue.cc:237] T c6f8a2b1f923426e989e4d2dbcac9e2d P a3e62ccad691413ea2554182f73418a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:03:51.244228 30236 catalog_manager.cc:5484] T 55f4202b341f42eb8e132766b634263b P 99f6ca2d65d94b33b7a1d2341c24179a reported cstate change: term changed from 0 to 1, leader changed from <none> to 99f6ca2d65d94b33b7a1d2341c24179a (127.29.124.133). New cstate: current_term: 1 leader_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } health_report { overall_health: HEALTHY } } }
I20240501 14:03:51.246593 30902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd60b6c4ba4b4eb58326755b08efb0e9" candidate_uuid: "870fbdcfdac84bacae0ad22dba25f626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" is_pre_election: true
I20240501 14:03:51.246966 30902 raft_consensus.cc:2454] T cd60b6c4ba4b4eb58326755b08efb0e9 P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 870fbdcfdac84bacae0ad22dba25f626 in term 0.
I20240501 14:03:51.247618 30706 leader_election.cc:302] T cd60b6c4ba4b4eb58326755b08efb0e9 P 870fbdcfdac84bacae0ad22dba25f626 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 870fbdcfdac84bacae0ad22dba25f626, 99f6ca2d65d94b33b7a1d2341c24179a; no voters:
I20240501 14:03:51.249657 31119 raft_consensus.cc:2781] T cd60b6c4ba4b4eb58326755b08efb0e9 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:03:51.249842 31119 raft_consensus.cc:482] T cd60b6c4ba4b4eb58326755b08efb0e9 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:03:51.249945 31119 raft_consensus.cc:3037] T cd60b6c4ba4b4eb58326755b08efb0e9 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:51.250671 30236 catalog_manager.cc:5484] T c6f8a2b1f923426e989e4d2dbcac9e2d P a3e62ccad691413ea2554182f73418a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to a3e62ccad691413ea2554182f73418a3 (127.29.124.131). New cstate: current_term: 1 leader_uuid: "a3e62ccad691413ea2554182f73418a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } health_report { overall_health: HEALTHY } } }
I20240501 14:03:51.252174 31119 raft_consensus.cc:504] T cd60b6c4ba4b4eb58326755b08efb0e9 P 870fbdcfdac84bacae0ad22dba25f626 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:03:51.252092 30509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd60b6c4ba4b4eb58326755b08efb0e9" candidate_uuid: "870fbdcfdac84bacae0ad22dba25f626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8883498098f4454bbfe563fb83d31899" is_pre_election: true
I20240501 14:03:51.252601 30509 raft_consensus.cc:2454] T cd60b6c4ba4b4eb58326755b08efb0e9 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 870fbdcfdac84bacae0ad22dba25f626 in term 0.
I20240501 14:03:51.252846 31119 leader_election.cc:288] T cd60b6c4ba4b4eb58326755b08efb0e9 P 870fbdcfdac84bacae0ad22dba25f626 [CANDIDATE]: Term 1 election: Requested vote from peers 99f6ca2d65d94b33b7a1d2341c24179a (127.29.124.133:35195), 8883498098f4454bbfe563fb83d31899 (127.29.124.130:43099)
I20240501 14:03:51.253387 30902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd60b6c4ba4b4eb58326755b08efb0e9" candidate_uuid: "870fbdcfdac84bacae0ad22dba25f626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99f6ca2d65d94b33b7a1d2341c24179a"
I20240501 14:03:51.253463 30509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cd60b6c4ba4b4eb58326755b08efb0e9" candidate_uuid: "870fbdcfdac84bacae0ad22dba25f626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8883498098f4454bbfe563fb83d31899"
I20240501 14:03:51.253577 30902 raft_consensus.cc:3037] T cd60b6c4ba4b4eb58326755b08efb0e9 P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:51.253629 30509 raft_consensus.cc:3037] T cd60b6c4ba4b4eb58326755b08efb0e9 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:51.255736 30902 raft_consensus.cc:2454] T cd60b6c4ba4b4eb58326755b08efb0e9 P 99f6ca2d65d94b33b7a1d2341c24179a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 870fbdcfdac84bacae0ad22dba25f626 in term 1.
I20240501 14:03:51.255834 30509 raft_consensus.cc:2454] T cd60b6c4ba4b4eb58326755b08efb0e9 P 8883498098f4454bbfe563fb83d31899 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 870fbdcfdac84bacae0ad22dba25f626 in term 1.
I20240501 14:03:51.256127 30706 leader_election.cc:302] T cd60b6c4ba4b4eb58326755b08efb0e9 P 870fbdcfdac84bacae0ad22dba25f626 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 870fbdcfdac84bacae0ad22dba25f626, 99f6ca2d65d94b33b7a1d2341c24179a; no voters:
I20240501 14:03:51.256424 31119 raft_consensus.cc:2781] T cd60b6c4ba4b4eb58326755b08efb0e9 P 870fbdcfdac84bacae0ad22dba25f626 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:03:51.256785 31119 raft_consensus.cc:686] T cd60b6c4ba4b4eb58326755b08efb0e9 P 870fbdcfdac84bacae0ad22dba25f626 [term 1 LEADER]: Becoming Leader. State: Replica: 870fbdcfdac84bacae0ad22dba25f626, State: Running, Role: LEADER
I20240501 14:03:51.257220 31119 consensus_queue.cc:237] T cd60b6c4ba4b4eb58326755b08efb0e9 P 870fbdcfdac84bacae0ad22dba25f626 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:03:51.261209 30236 catalog_manager.cc:5484] T cd60b6c4ba4b4eb58326755b08efb0e9 P 870fbdcfdac84bacae0ad22dba25f626 reported cstate change: term changed from 0 to 1, leader changed from <none> to 870fbdcfdac84bacae0ad22dba25f626 (127.29.124.132). New cstate: current_term: 1 leader_uuid: "870fbdcfdac84bacae0ad22dba25f626" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } health_report { overall_health: UNKNOWN } } }
W20240501 14:03:51.275861 30686 tablet.cc:2369] T 55f4202b341f42eb8e132766b634263b P a3e62ccad691413ea2554182f73418a3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 14:03:51.321041 31079 tablet.cc:2369] T c6f8a2b1f923426e989e4d2dbcac9e2d P 9964c1c76dd14f7dbce766b0c022ab51: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 14:03:51.646003 31138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:03:51.646466 31138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 14:03:51.649021 31116 consensus_queue.cc:1035] T 55f4202b341f42eb8e132766b634263b P 99f6ca2d65d94b33b7a1d2341c24179a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:03:51.658749 31138 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:03:51.659708 31146 consensus_queue.cc:1035] T 55f4202b341f42eb8e132766b634263b P 99f6ca2d65d94b33b7a1d2341c24179a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'
I20240501 14:03:51.752607 31119 consensus_queue.cc:1035] T cd60b6c4ba4b4eb58326755b08efb0e9 P 870fbdcfdac84bacae0ad22dba25f626 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:51.782487 31114 consensus_queue.cc:1035] T c6f8a2b1f923426e989e4d2dbcac9e2d P a3e62ccad691413ea2554182f73418a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:51.807514 31114 consensus_queue.cc:1035] T c6f8a2b1f923426e989e4d2dbcac9e2d P a3e62ccad691413ea2554182f73418a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:51.815519 31172 consensus_queue.cc:1035] T cd60b6c4ba4b4eb58326755b08efb0e9 P 870fbdcfdac84bacae0ad22dba25f626 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:03:51.869138 31163 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTableLocations -> {remote=127.29.124.190:35979, user_credentials={real_user=slave}} blocked reactor thread for 44843.7us
INSERT report
rows total: 2000
time total: 357.274 ms
time per row: 0.178637 ms
I20240501 14:03:52.157200 30233 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35960:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:03:52.165762 30489 tablet_service.cc:1467] Processing CreateTablet for tablet aa94dbe4c6bc47618980d9dfa4d34333 (DEFAULT_TABLE table=rebalance_test_table_1 [id=53467c77807848a4ab87eb475189ee60]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:52.165979 31013 tablet_service.cc:1467] Processing CreateTablet for tablet aa94dbe4c6bc47618980d9dfa4d34333 (DEFAULT_TABLE table=rebalance_test_table_1 [id=53467c77807848a4ab87eb475189ee60]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:52.166239 30489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa94dbe4c6bc47618980d9dfa4d34333. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:52.166357 31013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa94dbe4c6bc47618980d9dfa4d34333. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:52.167269 30881 tablet_service.cc:1467] Processing CreateTablet for tablet 3e7dddb0e5694c83acde0ee8063c640c (DEFAULT_TABLE table=rebalance_test_table_1 [id=53467c77807848a4ab87eb475189ee60]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:52.167668 30881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e7dddb0e5694c83acde0ee8063c640c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:52.172010 31211 tablet_bootstrap.cc:492] T 3e7dddb0e5694c83acde0ee8063c640c P 99f6ca2d65d94b33b7a1d2341c24179a: Bootstrap starting.
I20240501 14:03:52.172510 30751 tablet_service.cc:1467] Processing CreateTablet for tablet aa94dbe4c6bc47618980d9dfa4d34333 (DEFAULT_TABLE table=rebalance_test_table_1 [id=53467c77807848a4ab87eb475189ee60]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:52.172650 30750 tablet_service.cc:1467] Processing CreateTablet for tablet 1b78541b2c4b47cca4e5f22ec9ffd5cd (DEFAULT_TABLE table=rebalance_test_table_1 [id=53467c77807848a4ab87eb475189ee60]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:52.172923 30751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet aa94dbe4c6bc47618980d9dfa4d34333. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:52.173157 30750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b78541b2c4b47cca4e5f22ec9ffd5cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:52.173902 31211 tablet_bootstrap.cc:654] T 3e7dddb0e5694c83acde0ee8063c640c P 99f6ca2d65d94b33b7a1d2341c24179a: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:52.174302 30619 tablet_service.cc:1467] Processing CreateTablet for tablet 3e7dddb0e5694c83acde0ee8063c640c (DEFAULT_TABLE table=rebalance_test_table_1 [id=53467c77807848a4ab87eb475189ee60]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:52.174687 30619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e7dddb0e5694c83acde0ee8063c640c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:52.177932 30358 tablet_service.cc:1467] Processing CreateTablet for tablet 3e7dddb0e5694c83acde0ee8063c640c (DEFAULT_TABLE table=rebalance_test_table_1 [id=53467c77807848a4ab87eb475189ee60]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:52.178022 31012 tablet_service.cc:1467] Processing CreateTablet for tablet 1b78541b2c4b47cca4e5f22ec9ffd5cd (DEFAULT_TABLE table=rebalance_test_table_1 [id=53467c77807848a4ab87eb475189ee60]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:52.178373 30358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e7dddb0e5694c83acde0ee8063c640c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:52.178413 31012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b78541b2c4b47cca4e5f22ec9ffd5cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:52.181455 31214 tablet_bootstrap.cc:492] T aa94dbe4c6bc47618980d9dfa4d34333 P 9964c1c76dd14f7dbce766b0c022ab51: Bootstrap starting.
I20240501 14:03:52.181704 30488 tablet_service.cc:1467] Processing CreateTablet for tablet 1b78541b2c4b47cca4e5f22ec9ffd5cd (DEFAULT_TABLE table=rebalance_test_table_1 [id=53467c77807848a4ab87eb475189ee60]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:52.182152 30488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b78541b2c4b47cca4e5f22ec9ffd5cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:52.183284 31214 tablet_bootstrap.cc:654] T aa94dbe4c6bc47618980d9dfa4d34333 P 9964c1c76dd14f7dbce766b0c022ab51: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:52.189177 31212 tablet_bootstrap.cc:492] T aa94dbe4c6bc47618980d9dfa4d34333 P 8883498098f4454bbfe563fb83d31899: Bootstrap starting.
I20240501 14:03:52.191640 31212 tablet_bootstrap.cc:654] T aa94dbe4c6bc47618980d9dfa4d34333 P 8883498098f4454bbfe563fb83d31899: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:52.197139 31216 tablet_bootstrap.cc:492] T 3e7dddb0e5694c83acde0ee8063c640c P c0fad54e965647aaaa34b56674646cb4: Bootstrap starting.
I20240501 14:03:52.199103 31216 tablet_bootstrap.cc:654] T 3e7dddb0e5694c83acde0ee8063c640c P c0fad54e965647aaaa34b56674646cb4: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:52.201586 31212 tablet_bootstrap.cc:492] T aa94dbe4c6bc47618980d9dfa4d34333 P 8883498098f4454bbfe563fb83d31899: No bootstrap required, opened a new log
I20240501 14:03:52.202339 31212 ts_tablet_manager.cc:1367] T aa94dbe4c6bc47618980d9dfa4d34333 P 8883498098f4454bbfe563fb83d31899: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20240501 14:03:52.203593 31212 raft_consensus.cc:348] T aa94dbe4c6bc47618980d9dfa4d34333 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:03:52.205160 31212 raft_consensus.cc:374] T aa94dbe4c6bc47618980d9dfa4d34333 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:52.205446 31212 raft_consensus.cc:729] T aa94dbe4c6bc47618980d9dfa4d34333 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8883498098f4454bbfe563fb83d31899, State: Initialized, Role: FOLLOWER
I20240501 14:03:52.205961 31212 consensus_queue.cc:260] T aa94dbe4c6bc47618980d9dfa4d34333 P 8883498098f4454bbfe563fb83d31899 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:03:52.207814 31212 ts_tablet_manager.cc:1398] T aa94dbe4c6bc47618980d9dfa4d34333 P 8883498098f4454bbfe563fb83d31899: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:03:52.208505 31212 tablet_bootstrap.cc:492] T 1b78541b2c4b47cca4e5f22ec9ffd5cd P 8883498098f4454bbfe563fb83d31899: Bootstrap starting.
I20240501 14:03:52.213408 31212 tablet_bootstrap.cc:654] T 1b78541b2c4b47cca4e5f22ec9ffd5cd P 8883498098f4454bbfe563fb83d31899: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:52.216135 31215 tablet_bootstrap.cc:492] T 1b78541b2c4b47cca4e5f22ec9ffd5cd P 870fbdcfdac84bacae0ad22dba25f626: Bootstrap starting.
I20240501 14:03:52.217846 31220 tablet_bootstrap.cc:492] T 3e7dddb0e5694c83acde0ee8063c640c P a3e62ccad691413ea2554182f73418a3: Bootstrap starting.
I20240501 14:03:52.219244 31215 tablet_bootstrap.cc:654] T 1b78541b2c4b47cca4e5f22ec9ffd5cd P 870fbdcfdac84bacae0ad22dba25f626: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:52.219363 31214 tablet_bootstrap.cc:492] T aa94dbe4c6bc47618980d9dfa4d34333 P 9964c1c76dd14f7dbce766b0c022ab51: No bootstrap required, opened a new log
I20240501 14:03:52.219533 31214 ts_tablet_manager.cc:1367] T aa94dbe4c6bc47618980d9dfa4d34333 P 9964c1c76dd14f7dbce766b0c022ab51: Time spent bootstrapping tablet: real 0.038s user 0.014s sys 0.018s
I20240501 14:03:52.220623 31214 raft_consensus.cc:348] T aa94dbe4c6bc47618980d9dfa4d34333 P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:03:52.220916 31214 raft_consensus.cc:374] T aa94dbe4c6bc47618980d9dfa4d34333 P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:52.221017 31214 raft_consensus.cc:729] T aa94dbe4c6bc47618980d9dfa4d34333 P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9964c1c76dd14f7dbce766b0c022ab51, State: Initialized, Role: FOLLOWER
I20240501 14:03:52.221310 31214 consensus_queue.cc:260] T aa94dbe4c6bc47618980d9dfa4d34333 P 9964c1c76dd14f7dbce766b0c022ab51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:03:52.221904 31211 tablet_bootstrap.cc:492] T 3e7dddb0e5694c83acde0ee8063c640c P 99f6ca2d65d94b33b7a1d2341c24179a: No bootstrap required, opened a new log
I20240501 14:03:52.222071 31214 ts_tablet_manager.cc:1398] T aa94dbe4c6bc47618980d9dfa4d34333 P 9964c1c76dd14f7dbce766b0c022ab51: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 14:03:52.222101 31211 ts_tablet_manager.cc:1367] T 3e7dddb0e5694c83acde0ee8063c640c P 99f6ca2d65d94b33b7a1d2341c24179a: Time spent bootstrapping tablet: real 0.050s user 0.003s sys 0.012s
I20240501 14:03:52.222359 31214 tablet_bootstrap.cc:492] T 1b78541b2c4b47cca4e5f22ec9ffd5cd P 9964c1c76dd14f7dbce766b0c022ab51: Bootstrap starting.
I20240501 14:03:52.223066 31211 raft_consensus.cc:348] T 3e7dddb0e5694c83acde0ee8063c640c P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } }
I20240501 14:03:52.223315 31211 raft_consensus.cc:374] T 3e7dddb0e5694c83acde0ee8063c640c P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:52.223451 31211 raft_consensus.cc:729] T 3e7dddb0e5694c83acde0ee8063c640c P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99f6ca2d65d94b33b7a1d2341c24179a, State: Initialized, Role: FOLLOWER
I20240501 14:03:52.223863 31214 tablet_bootstrap.cc:654] T 1b78541b2c4b47cca4e5f22ec9ffd5cd P 9964c1c76dd14f7dbce766b0c022ab51: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:52.223752 31211 consensus_queue.cc:260] T 3e7dddb0e5694c83acde0ee8063c640c P 99f6ca2d65d94b33b7a1d2341c24179a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } }
I20240501 14:03:52.224489 31211 ts_tablet_manager.cc:1398] T 3e7dddb0e5694c83acde0ee8063c640c P 99f6ca2d65d94b33b7a1d2341c24179a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:03:52.225219 31220 tablet_bootstrap.cc:654] T 3e7dddb0e5694c83acde0ee8063c640c P a3e62ccad691413ea2554182f73418a3: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:52.227278 31216 tablet_bootstrap.cc:492] T 3e7dddb0e5694c83acde0ee8063c640c P c0fad54e965647aaaa34b56674646cb4: No bootstrap required, opened a new log
I20240501 14:03:52.227479 31216 ts_tablet_manager.cc:1367] T 3e7dddb0e5694c83acde0ee8063c640c P c0fad54e965647aaaa34b56674646cb4: Time spent bootstrapping tablet: real 0.030s user 0.005s sys 0.000s
I20240501 14:03:52.228333 31216 raft_consensus.cc:348] T 3e7dddb0e5694c83acde0ee8063c640c P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } }
I20240501 14:03:52.229689 31216 raft_consensus.cc:374] T 3e7dddb0e5694c83acde0ee8063c640c P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:52.229852 31216 raft_consensus.cc:729] T 3e7dddb0e5694c83acde0ee8063c640c P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0fad54e965647aaaa34b56674646cb4, State: Initialized, Role: FOLLOWER
I20240501 14:03:52.230255 31216 consensus_queue.cc:260] T 3e7dddb0e5694c83acde0ee8063c640c P c0fad54e965647aaaa34b56674646cb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } }
I20240501 14:03:52.231441 31214 tablet_bootstrap.cc:492] T 1b78541b2c4b47cca4e5f22ec9ffd5cd P 9964c1c76dd14f7dbce766b0c022ab51: No bootstrap required, opened a new log
I20240501 14:03:52.231451 31212 tablet_bootstrap.cc:492] T 1b78541b2c4b47cca4e5f22ec9ffd5cd P 8883498098f4454bbfe563fb83d31899: No bootstrap required, opened a new log
I20240501 14:03:52.231593 31212 ts_tablet_manager.cc:1367] T 1b78541b2c4b47cca4e5f22ec9ffd5cd P 8883498098f4454bbfe563fb83d31899: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.016s
I20240501 14:03:52.231593 31214 ts_tablet_manager.cc:1367] T 1b78541b2c4b47cca4e5f22ec9ffd5cd P 9964c1c76dd14f7dbce766b0c022ab51: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20240501 14:03:52.232298 31212 raft_consensus.cc:348] T 1b78541b2c4b47cca4e5f22ec9ffd5cd P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:03:52.232508 31212 raft_consensus.cc:374] T 1b78541b2c4b47cca4e5f22ec9ffd5cd P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:52.232576 31212 raft_consensus.cc:729] T 1b78541b2c4b47cca4e5f22ec9ffd5cd P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8883498098f4454bbfe563fb83d31899, State: Initialized, Role: FOLLOWER
I20240501 14:03:52.232612 31214 raft_consensus.cc:348] T 1b78541b2c4b47cca4e5f22ec9ffd5cd P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:03:52.232793 31214 raft_consensus.cc:374] T 1b78541b2c4b47cca4e5f22ec9ffd5cd P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:52.232859 31214 raft_consensus.cc:729] T 1b78541b2c4b47cca4e5f22ec9ffd5cd P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9964c1c76dd14f7dbce766b0c022ab51, State: Initialized, Role: FOLLOWER
I20240501 14:03:52.232842 31212 consensus_queue.cc:260] T 1b78541b2c4b47cca4e5f22ec9ffd5cd P 8883498098f4454bbfe563fb83d31899 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:03:52.233081 31215 tablet_bootstrap.cc:492] T 1b78541b2c4b47cca4e5f22ec9ffd5cd P 870fbdcfdac84bacae0ad22dba25f626: No bootstrap required, opened a new log
I20240501 14:03:52.233229 31215 ts_tablet_manager.cc:1367] T 1b78541b2c4b47cca4e5f22ec9ffd5cd P 870fbdcfdac84bacae0ad22dba25f626: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.003s
I20240501 14:03:52.233114 31214 consensus_queue.cc:260] T 1b78541b2c4b47cca4e5f22ec9ffd5cd P 9964c1c76dd14f7dbce766b0c022ab51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:03:52.233628 31214 ts_tablet_manager.cc:1398] T 1b78541b2c4b47cca4e5f22ec9ffd5cd P 9964c1c76dd14f7dbce766b0c022ab51: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:03:52.234215 31215 raft_consensus.cc:348] T 1b78541b2c4b47cca4e5f22ec9ffd5cd P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:03:52.234445 31215 raft_consensus.cc:374] T 1b78541b2c4b47cca4e5f22ec9ffd5cd P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:52.234529 31215 raft_consensus.cc:729] T 1b78541b2c4b47cca4e5f22ec9ffd5cd P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 870fbdcfdac84bacae0ad22dba25f626, State: Initialized, Role: FOLLOWER
I20240501 14:03:52.234745 31216 ts_tablet_manager.cc:1398] T 3e7dddb0e5694c83acde0ee8063c640c P c0fad54e965647aaaa34b56674646cb4: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:03:52.234767 31215 consensus_queue.cc:260] T 1b78541b2c4b47cca4e5f22ec9ffd5cd P 870fbdcfdac84bacae0ad22dba25f626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:03:52.236907 31215 ts_tablet_manager.cc:1398] T 1b78541b2c4b47cca4e5f22ec9ffd5cd P 870fbdcfdac84bacae0ad22dba25f626: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:03:52.237203 31215 tablet_bootstrap.cc:492] T aa94dbe4c6bc47618980d9dfa4d34333 P 870fbdcfdac84bacae0ad22dba25f626: Bootstrap starting.
I20240501 14:03:52.237509 31212 ts_tablet_manager.cc:1398] T 1b78541b2c4b47cca4e5f22ec9ffd5cd P 8883498098f4454bbfe563fb83d31899: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240501 14:03:52.239274 31215 tablet_bootstrap.cc:654] T aa94dbe4c6bc47618980d9dfa4d34333 P 870fbdcfdac84bacae0ad22dba25f626: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:52.239708 31220 tablet_bootstrap.cc:492] T 3e7dddb0e5694c83acde0ee8063c640c P a3e62ccad691413ea2554182f73418a3: No bootstrap required, opened a new log
I20240501 14:03:52.239866 31220 ts_tablet_manager.cc:1367] T 3e7dddb0e5694c83acde0ee8063c640c P a3e62ccad691413ea2554182f73418a3: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.000s
I20240501 14:03:52.240694 31220 raft_consensus.cc:348] T 3e7dddb0e5694c83acde0ee8063c640c P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } }
I20240501 14:03:52.240908 31220 raft_consensus.cc:374] T 3e7dddb0e5694c83acde0ee8063c640c P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:52.241005 31220 raft_consensus.cc:729] T 3e7dddb0e5694c83acde0ee8063c640c P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3e62ccad691413ea2554182f73418a3, State: Initialized, Role: FOLLOWER
I20240501 14:03:52.241268 31220 consensus_queue.cc:260] T 3e7dddb0e5694c83acde0ee8063c640c P a3e62ccad691413ea2554182f73418a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } }
I20240501 14:03:52.241919 31220 ts_tablet_manager.cc:1398] T 3e7dddb0e5694c83acde0ee8063c640c P a3e62ccad691413ea2554182f73418a3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:03:52.243894 31215 tablet_bootstrap.cc:492] T aa94dbe4c6bc47618980d9dfa4d34333 P 870fbdcfdac84bacae0ad22dba25f626: No bootstrap required, opened a new log
I20240501 14:03:52.244004 31199 raft_consensus.cc:482] T 1b78541b2c4b47cca4e5f22ec9ffd5cd P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:03:52.244094 31215 ts_tablet_manager.cc:1367] T aa94dbe4c6bc47618980d9dfa4d34333 P 870fbdcfdac84bacae0ad22dba25f626: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20240501 14:03:52.244131 31199 raft_consensus.cc:504] T 1b78541b2c4b47cca4e5f22ec9ffd5cd P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:03:52.245069 31215 raft_consensus.cc:348] T aa94dbe4c6bc47618980d9dfa4d34333 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:03:52.245301 31215 raft_consensus.cc:374] T aa94dbe4c6bc47618980d9dfa4d34333 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:52.245373 31199 leader_election.cc:288] T 1b78541b2c4b47cca4e5f22ec9ffd5cd P 870fbdcfdac84bacae0ad22dba25f626 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9964c1c76dd14f7dbce766b0c022ab51 (127.29.124.134:38101), 8883498098f4454bbfe563fb83d31899 (127.29.124.130:43099)
I20240501 14:03:52.245399 31215 raft_consensus.cc:729] T aa94dbe4c6bc47618980d9dfa4d34333 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 870fbdcfdac84bacae0ad22dba25f626, State: Initialized, Role: FOLLOWER
I20240501 14:03:52.245689 31215 consensus_queue.cc:260] T aa94dbe4c6bc47618980d9dfa4d34333 P 870fbdcfdac84bacae0ad22dba25f626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:03:52.245975 30508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b78541b2c4b47cca4e5f22ec9ffd5cd" candidate_uuid: "870fbdcfdac84bacae0ad22dba25f626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8883498098f4454bbfe563fb83d31899" is_pre_election: true
I20240501 14:03:52.246176 31215 ts_tablet_manager.cc:1398] T aa94dbe4c6bc47618980d9dfa4d34333 P 870fbdcfdac84bacae0ad22dba25f626: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:03:52.246210 30508 raft_consensus.cc:2454] T 1b78541b2c4b47cca4e5f22ec9ffd5cd P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 870fbdcfdac84bacae0ad22dba25f626 in term 0.
I20240501 14:03:52.246591 30707 leader_election.cc:302] T 1b78541b2c4b47cca4e5f22ec9ffd5cd P 870fbdcfdac84bacae0ad22dba25f626 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 870fbdcfdac84bacae0ad22dba25f626, 8883498098f4454bbfe563fb83d31899; no voters:
I20240501 14:03:52.246824 31199 raft_consensus.cc:2781] T 1b78541b2c4b47cca4e5f22ec9ffd5cd P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:03:52.246937 31199 raft_consensus.cc:482] T 1b78541b2c4b47cca4e5f22ec9ffd5cd P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:03:52.247033 31199 raft_consensus.cc:3037] T 1b78541b2c4b47cca4e5f22ec9ffd5cd P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:52.249913 31199 raft_consensus.cc:504] T 1b78541b2c4b47cca4e5f22ec9ffd5cd P 870fbdcfdac84bacae0ad22dba25f626 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:03:52.250553 31199 leader_election.cc:288] T 1b78541b2c4b47cca4e5f22ec9ffd5cd P 870fbdcfdac84bacae0ad22dba25f626 [CANDIDATE]: Term 1 election: Requested vote from peers 9964c1c76dd14f7dbce766b0c022ab51 (127.29.124.134:38101), 8883498098f4454bbfe563fb83d31899 (127.29.124.130:43099)
I20240501 14:03:52.251083 30508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b78541b2c4b47cca4e5f22ec9ffd5cd" candidate_uuid: "870fbdcfdac84bacae0ad22dba25f626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8883498098f4454bbfe563fb83d31899"
I20240501 14:03:52.251276 30508 raft_consensus.cc:3037] T 1b78541b2c4b47cca4e5f22ec9ffd5cd P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:52.253638 30508 raft_consensus.cc:2454] T 1b78541b2c4b47cca4e5f22ec9ffd5cd P 8883498098f4454bbfe563fb83d31899 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 870fbdcfdac84bacae0ad22dba25f626 in term 1.
I20240501 14:03:52.253994 30707 leader_election.cc:302] T 1b78541b2c4b47cca4e5f22ec9ffd5cd P 870fbdcfdac84bacae0ad22dba25f626 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 870fbdcfdac84bacae0ad22dba25f626, 8883498098f4454bbfe563fb83d31899; no voters:
I20240501 14:03:52.254256 31199 raft_consensus.cc:2781] T 1b78541b2c4b47cca4e5f22ec9ffd5cd P 870fbdcfdac84bacae0ad22dba25f626 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:03:52.254379 31199 raft_consensus.cc:686] T 1b78541b2c4b47cca4e5f22ec9ffd5cd P 870fbdcfdac84bacae0ad22dba25f626 [term 1 LEADER]: Becoming Leader. State: Replica: 870fbdcfdac84bacae0ad22dba25f626, State: Running, Role: LEADER
I20240501 14:03:52.254611 31199 consensus_queue.cc:237] T 1b78541b2c4b47cca4e5f22ec9ffd5cd P 870fbdcfdac84bacae0ad22dba25f626 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:03:52.257005 30229 catalog_manager.cc:5484] T 1b78541b2c4b47cca4e5f22ec9ffd5cd P 870fbdcfdac84bacae0ad22dba25f626 reported cstate change: term changed from 0 to 1, leader changed from <none> to 870fbdcfdac84bacae0ad22dba25f626 (127.29.124.132). New cstate: current_term: 1 leader_uuid: "870fbdcfdac84bacae0ad22dba25f626" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } health_report { overall_health: HEALTHY } } }
I20240501 14:03:52.263073 31032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b78541b2c4b47cca4e5f22ec9ffd5cd" candidate_uuid: "870fbdcfdac84bacae0ad22dba25f626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9964c1c76dd14f7dbce766b0c022ab51" is_pre_election: true
I20240501 14:03:52.263165 31033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b78541b2c4b47cca4e5f22ec9ffd5cd" candidate_uuid: "870fbdcfdac84bacae0ad22dba25f626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9964c1c76dd14f7dbce766b0c022ab51"
I20240501 14:03:52.263269 31032 raft_consensus.cc:2454] T 1b78541b2c4b47cca4e5f22ec9ffd5cd P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 870fbdcfdac84bacae0ad22dba25f626 in term 0.
I20240501 14:03:52.309190 31130 raft_consensus.cc:482] T 3e7dddb0e5694c83acde0ee8063c640c P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:03:52.309386 31130 raft_consensus.cc:504] T 3e7dddb0e5694c83acde0ee8063c640c P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } }
I20240501 14:03:52.309993 31130 leader_election.cc:288] T 3e7dddb0e5694c83acde0ee8063c640c P 99f6ca2d65d94b33b7a1d2341c24179a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3e62ccad691413ea2554182f73418a3 (127.29.124.131:43965), c0fad54e965647aaaa34b56674646cb4 (127.29.124.129:42363)
I20240501 14:03:52.310482 30638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e7dddb0e5694c83acde0ee8063c640c" candidate_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3e62ccad691413ea2554182f73418a3" is_pre_election: true
I20240501 14:03:52.310514 30377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e7dddb0e5694c83acde0ee8063c640c" candidate_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0fad54e965647aaaa34b56674646cb4" is_pre_election: true
I20240501 14:03:52.310701 30638 raft_consensus.cc:2454] T 3e7dddb0e5694c83acde0ee8063c640c P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99f6ca2d65d94b33b7a1d2341c24179a in term 0.
I20240501 14:03:52.310714 30377 raft_consensus.cc:2454] T 3e7dddb0e5694c83acde0ee8063c640c P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99f6ca2d65d94b33b7a1d2341c24179a in term 0.
I20240501 14:03:52.311110 30835 leader_election.cc:302] T 3e7dddb0e5694c83acde0ee8063c640c P 99f6ca2d65d94b33b7a1d2341c24179a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99f6ca2d65d94b33b7a1d2341c24179a, a3e62ccad691413ea2554182f73418a3; no voters:
I20240501 14:03:52.311362 31130 raft_consensus.cc:2781] T 3e7dddb0e5694c83acde0ee8063c640c P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:03:52.311455 31130 raft_consensus.cc:482] T 3e7dddb0e5694c83acde0ee8063c640c P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:03:52.311515 31130 raft_consensus.cc:3037] T 3e7dddb0e5694c83acde0ee8063c640c P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:52.313304 31130 raft_consensus.cc:504] T 3e7dddb0e5694c83acde0ee8063c640c P 99f6ca2d65d94b33b7a1d2341c24179a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } }
I20240501 14:03:52.313771 31130 leader_election.cc:288] T 3e7dddb0e5694c83acde0ee8063c640c P 99f6ca2d65d94b33b7a1d2341c24179a [CANDIDATE]: Term 1 election: Requested vote from peers a3e62ccad691413ea2554182f73418a3 (127.29.124.131:43965), c0fad54e965647aaaa34b56674646cb4 (127.29.124.129:42363)
I20240501 14:03:52.314201 30638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e7dddb0e5694c83acde0ee8063c640c" candidate_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3e62ccad691413ea2554182f73418a3"
I20240501 14:03:52.314357 30638 raft_consensus.cc:3037] T 3e7dddb0e5694c83acde0ee8063c640c P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:52.314296 30377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e7dddb0e5694c83acde0ee8063c640c" candidate_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0fad54e965647aaaa34b56674646cb4"
I20240501 14:03:52.314473 30377 raft_consensus.cc:3037] T 3e7dddb0e5694c83acde0ee8063c640c P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:52.316459 30638 raft_consensus.cc:2454] T 3e7dddb0e5694c83acde0ee8063c640c P a3e62ccad691413ea2554182f73418a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99f6ca2d65d94b33b7a1d2341c24179a in term 1.
I20240501 14:03:52.316591 30377 raft_consensus.cc:2454] T 3e7dddb0e5694c83acde0ee8063c640c P c0fad54e965647aaaa34b56674646cb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99f6ca2d65d94b33b7a1d2341c24179a in term 1.
I20240501 14:03:52.316833 30835 leader_election.cc:302] T 3e7dddb0e5694c83acde0ee8063c640c P 99f6ca2d65d94b33b7a1d2341c24179a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99f6ca2d65d94b33b7a1d2341c24179a, a3e62ccad691413ea2554182f73418a3; no voters:
I20240501 14:03:52.317087 31130 raft_consensus.cc:2781] T 3e7dddb0e5694c83acde0ee8063c640c P 99f6ca2d65d94b33b7a1d2341c24179a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:03:52.317222 31130 raft_consensus.cc:686] T 3e7dddb0e5694c83acde0ee8063c640c P 99f6ca2d65d94b33b7a1d2341c24179a [term 1 LEADER]: Becoming Leader. State: Replica: 99f6ca2d65d94b33b7a1d2341c24179a, State: Running, Role: LEADER
I20240501 14:03:52.317534 31130 consensus_queue.cc:237] T 3e7dddb0e5694c83acde0ee8063c640c P 99f6ca2d65d94b33b7a1d2341c24179a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } }
I20240501 14:03:52.320025 30229 catalog_manager.cc:5484] T 3e7dddb0e5694c83acde0ee8063c640c P 99f6ca2d65d94b33b7a1d2341c24179a reported cstate change: term changed from 0 to 1, leader changed from <none> to 99f6ca2d65d94b33b7a1d2341c24179a (127.29.124.133). New cstate: current_term: 1 leader_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } health_report { overall_health: UNKNOWN } } }
I20240501 14:03:52.354509 31168 raft_consensus.cc:482] T aa94dbe4c6bc47618980d9dfa4d34333 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:03:52.354671 31168 raft_consensus.cc:504] T aa94dbe4c6bc47618980d9dfa4d34333 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:03:52.356231 31168 leader_election.cc:288] T aa94dbe4c6bc47618980d9dfa4d34333 P 8883498098f4454bbfe563fb83d31899 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9964c1c76dd14f7dbce766b0c022ab51 (127.29.124.134:38101), 870fbdcfdac84bacae0ad22dba25f626 (127.29.124.132:44267)
I20240501 14:03:52.362469 31199 raft_consensus.cc:482] T aa94dbe4c6bc47618980d9dfa4d34333 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:03:52.362694 31199 raft_consensus.cc:504] T aa94dbe4c6bc47618980d9dfa4d34333 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:03:52.363348 31199 leader_election.cc:288] T aa94dbe4c6bc47618980d9dfa4d34333 P 870fbdcfdac84bacae0ad22dba25f626 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8883498098f4454bbfe563fb83d31899 (127.29.124.130:43099), 9964c1c76dd14f7dbce766b0c022ab51 (127.29.124.134:38101)
I20240501 14:03:52.363807 30771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa94dbe4c6bc47618980d9dfa4d34333" candidate_uuid: "8883498098f4454bbfe563fb83d31899" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "870fbdcfdac84bacae0ad22dba25f626" is_pre_election: true
I20240501 14:03:52.364205 30771 raft_consensus.cc:2454] T aa94dbe4c6bc47618980d9dfa4d34333 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8883498098f4454bbfe563fb83d31899 in term 0.
I20240501 14:03:52.364179 30508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa94dbe4c6bc47618980d9dfa4d34333" candidate_uuid: "870fbdcfdac84bacae0ad22dba25f626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8883498098f4454bbfe563fb83d31899" is_pre_election: true
I20240501 14:03:52.364413 30508 raft_consensus.cc:2454] T aa94dbe4c6bc47618980d9dfa4d34333 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 870fbdcfdac84bacae0ad22dba25f626 in term 0.
I20240501 14:03:52.364493 31032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa94dbe4c6bc47618980d9dfa4d34333" candidate_uuid: "870fbdcfdac84bacae0ad22dba25f626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9964c1c76dd14f7dbce766b0c022ab51" is_pre_election: true
I20240501 14:03:52.364696 31032 raft_consensus.cc:2454] T aa94dbe4c6bc47618980d9dfa4d34333 P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 870fbdcfdac84bacae0ad22dba25f626 in term 0.
I20240501 14:03:52.365041 30704 leader_election.cc:302] T aa94dbe4c6bc47618980d9dfa4d34333 P 870fbdcfdac84bacae0ad22dba25f626 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 870fbdcfdac84bacae0ad22dba25f626, 9964c1c76dd14f7dbce766b0c022ab51; no voters:
I20240501 14:03:52.365015 30445 leader_election.cc:302] T aa94dbe4c6bc47618980d9dfa4d34333 P 8883498098f4454bbfe563fb83d31899 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 870fbdcfdac84bacae0ad22dba25f626, 8883498098f4454bbfe563fb83d31899; no voters:
I20240501 14:03:52.365223 31032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa94dbe4c6bc47618980d9dfa4d34333" candidate_uuid: "8883498098f4454bbfe563fb83d31899" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9964c1c76dd14f7dbce766b0c022ab51" is_pre_election: true
I20240501 14:03:52.365402 31168 raft_consensus.cc:2781] T aa94dbe4c6bc47618980d9dfa4d34333 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:03:52.365432 31032 raft_consensus.cc:2454] T aa94dbe4c6bc47618980d9dfa4d34333 P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8883498098f4454bbfe563fb83d31899 in term 0.
I20240501 14:03:52.365463 31199 raft_consensus.cc:2781] T aa94dbe4c6bc47618980d9dfa4d34333 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:03:52.365504 31168 raft_consensus.cc:482] T aa94dbe4c6bc47618980d9dfa4d34333 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:03:52.365537 31199 raft_consensus.cc:482] T aa94dbe4c6bc47618980d9dfa4d34333 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:03:52.365585 31168 raft_consensus.cc:3037] T aa94dbe4c6bc47618980d9dfa4d34333 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:52.365622 31199 raft_consensus.cc:3037] T aa94dbe4c6bc47618980d9dfa4d34333 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:52.368211 31199 raft_consensus.cc:504] T aa94dbe4c6bc47618980d9dfa4d34333 P 870fbdcfdac84bacae0ad22dba25f626 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:03:52.368304 31168 raft_consensus.cc:504] T aa94dbe4c6bc47618980d9dfa4d34333 P 8883498098f4454bbfe563fb83d31899 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:03:52.368849 31199 leader_election.cc:288] T aa94dbe4c6bc47618980d9dfa4d34333 P 870fbdcfdac84bacae0ad22dba25f626 [CANDIDATE]: Term 1 election: Requested vote from peers 8883498098f4454bbfe563fb83d31899 (127.29.124.130:43099), 9964c1c76dd14f7dbce766b0c022ab51 (127.29.124.134:38101)
I20240501 14:03:52.368964 31168 leader_election.cc:288] T aa94dbe4c6bc47618980d9dfa4d34333 P 8883498098f4454bbfe563fb83d31899 [CANDIDATE]: Term 1 election: Requested vote from peers 9964c1c76dd14f7dbce766b0c022ab51 (127.29.124.134:38101), 870fbdcfdac84bacae0ad22dba25f626 (127.29.124.132:44267)
I20240501 14:03:52.369233 30508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa94dbe4c6bc47618980d9dfa4d34333" candidate_uuid: "870fbdcfdac84bacae0ad22dba25f626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8883498098f4454bbfe563fb83d31899"
I20240501 14:03:52.369406 30771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa94dbe4c6bc47618980d9dfa4d34333" candidate_uuid: "8883498098f4454bbfe563fb83d31899" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "870fbdcfdac84bacae0ad22dba25f626"
I20240501 14:03:52.369434 31032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa94dbe4c6bc47618980d9dfa4d34333" candidate_uuid: "8883498098f4454bbfe563fb83d31899" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9964c1c76dd14f7dbce766b0c022ab51"
I20240501 14:03:52.369510 30508 raft_consensus.cc:2379] T aa94dbe4c6bc47618980d9dfa4d34333 P 8883498098f4454bbfe563fb83d31899 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 870fbdcfdac84bacae0ad22dba25f626 in current term 1: Already voted for candidate 8883498098f4454bbfe563fb83d31899 in this term.
I20240501 14:03:52.369539 31033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "aa94dbe4c6bc47618980d9dfa4d34333" candidate_uuid: "870fbdcfdac84bacae0ad22dba25f626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9964c1c76dd14f7dbce766b0c022ab51"
I20240501 14:03:52.369601 31032 raft_consensus.cc:3037] T aa94dbe4c6bc47618980d9dfa4d34333 P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:52.369632 30771 raft_consensus.cc:2379] T aa94dbe4c6bc47618980d9dfa4d34333 P 870fbdcfdac84bacae0ad22dba25f626 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 8883498098f4454bbfe563fb83d31899 in current term 1: Already voted for candidate 870fbdcfdac84bacae0ad22dba25f626 in this term.
I20240501 14:03:52.370340 30707 leader_election.cc:302] T aa94dbe4c6bc47618980d9dfa4d34333 P 870fbdcfdac84bacae0ad22dba25f626 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 870fbdcfdac84bacae0ad22dba25f626; no voters: 8883498098f4454bbfe563fb83d31899, 9964c1c76dd14f7dbce766b0c022ab51
I20240501 14:03:52.370576 31199 raft_consensus.cc:2726] T aa94dbe4c6bc47618980d9dfa4d34333 P 870fbdcfdac84bacae0ad22dba25f626 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240501 14:03:52.371671 31032 raft_consensus.cc:2454] T aa94dbe4c6bc47618980d9dfa4d34333 P 9964c1c76dd14f7dbce766b0c022ab51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8883498098f4454bbfe563fb83d31899 in term 1.
I20240501 14:03:52.371986 30442 leader_election.cc:302] T aa94dbe4c6bc47618980d9dfa4d34333 P 8883498098f4454bbfe563fb83d31899 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8883498098f4454bbfe563fb83d31899, 9964c1c76dd14f7dbce766b0c022ab51; no voters: 870fbdcfdac84bacae0ad22dba25f626
I20240501 14:03:52.372248 31168 raft_consensus.cc:2781] T aa94dbe4c6bc47618980d9dfa4d34333 P 8883498098f4454bbfe563fb83d31899 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:03:52.372627 31168 raft_consensus.cc:686] T aa94dbe4c6bc47618980d9dfa4d34333 P 8883498098f4454bbfe563fb83d31899 [term 1 LEADER]: Becoming Leader. State: Replica: 8883498098f4454bbfe563fb83d31899, State: Running, Role: LEADER
I20240501 14:03:52.373086 31168 consensus_queue.cc:237] T aa94dbe4c6bc47618980d9dfa4d34333 P 8883498098f4454bbfe563fb83d31899 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:03:52.376277 30229 catalog_manager.cc:5484] T aa94dbe4c6bc47618980d9dfa4d34333 P 8883498098f4454bbfe563fb83d31899 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8883498098f4454bbfe563fb83d31899 (127.29.124.130). New cstate: current_term: 1 leader_uuid: "8883498098f4454bbfe563fb83d31899" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } health_report { overall_health: UNKNOWN } } }
W20240501 14:03:52.699856 31234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:03:52.700165 31234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:03:52.707562 31234 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:03:52.728091 31146 consensus_queue.cc:1035] T 3e7dddb0e5694c83acde0ee8063c640c P 99f6ca2d65d94b33b7a1d2341c24179a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:52.734045 31130 consensus_queue.cc:1035] T 3e7dddb0e5694c83acde0ee8063c640c P 99f6ca2d65d94b33b7a1d2341c24179a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_1'
I20240501 14:03:52.775514 31199 consensus_queue.cc:1035] T 1b78541b2c4b47cca4e5f22ec9ffd5cd P 870fbdcfdac84bacae0ad22dba25f626 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:52.804385 31032 raft_consensus.cc:3037] T 1b78541b2c4b47cca4e5f22ec9ffd5cd P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:52.808032 31199 consensus_queue.cc:1035] T 1b78541b2c4b47cca4e5f22ec9ffd5cd P 870fbdcfdac84bacae0ad22dba25f626 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:52.813803 31168 consensus_queue.cc:1035] T aa94dbe4c6bc47618980d9dfa4d34333 P 8883498098f4454bbfe563fb83d31899 [LEADER]: Connected to new peer: Peer: permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:52.827080 31232 consensus_queue.cc:1035] T aa94dbe4c6bc47618980d9dfa4d34333 P 8883498098f4454bbfe563fb83d31899 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 294.815 ms
time per row: 0.147407 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240501 14:03:54.006867 30229 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35960:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:03:54.015568 30619 tablet_service.cc:1467] Processing CreateTablet for tablet 91e42d11fe064d28b457e1bddff845b2 (DEFAULT_TABLE table=rebalance_test_table_2 [id=618889a5c8d145b6b0d2f74d197b3272]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:54.015713 30881 tablet_service.cc:1467] Processing CreateTablet for tablet 91e42d11fe064d28b457e1bddff845b2 (DEFAULT_TABLE table=rebalance_test_table_2 [id=618889a5c8d145b6b0d2f74d197b3272]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:54.016007 30619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91e42d11fe064d28b457e1bddff845b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:54.016060 30881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91e42d11fe064d28b457e1bddff845b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:54.016391 31012 tablet_service.cc:1467] Processing CreateTablet for tablet cfaffe32f00e434faf2580b5fbba1fb5 (DEFAULT_TABLE table=rebalance_test_table_2 [id=618889a5c8d145b6b0d2f74d197b3272]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:54.016784 31012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfaffe32f00e434faf2580b5fbba1fb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:54.016922 30620 tablet_service.cc:1467] Processing CreateTablet for tablet 88537bae3b3c401086e601638bc42e43 (DEFAULT_TABLE table=rebalance_test_table_2 [id=618889a5c8d145b6b0d2f74d197b3272]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:54.017336 30620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88537bae3b3c401086e601638bc42e43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:54.017967 30357 tablet_service.cc:1467] Processing CreateTablet for tablet 88537bae3b3c401086e601638bc42e43 (DEFAULT_TABLE table=rebalance_test_table_2 [id=618889a5c8d145b6b0d2f74d197b3272]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:54.018358 30357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88537bae3b3c401086e601638bc42e43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:54.019829 30358 tablet_service.cc:1467] Processing CreateTablet for tablet 91e42d11fe064d28b457e1bddff845b2 (DEFAULT_TABLE table=rebalance_test_table_2 [id=618889a5c8d145b6b0d2f74d197b3272]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:54.020202 30358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91e42d11fe064d28b457e1bddff845b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:54.021839 30882 tablet_service.cc:1467] Processing CreateTablet for tablet 88537bae3b3c401086e601638bc42e43 (DEFAULT_TABLE table=rebalance_test_table_2 [id=618889a5c8d145b6b0d2f74d197b3272]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:54.022189 30882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88537bae3b3c401086e601638bc42e43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:54.026769 30750 tablet_service.cc:1467] Processing CreateTablet for tablet cfaffe32f00e434faf2580b5fbba1fb5 (DEFAULT_TABLE table=rebalance_test_table_2 [id=618889a5c8d145b6b0d2f74d197b3272]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:54.027209 30750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfaffe32f00e434faf2580b5fbba1fb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:54.029685 30488 tablet_service.cc:1467] Processing CreateTablet for tablet cfaffe32f00e434faf2580b5fbba1fb5 (DEFAULT_TABLE table=rebalance_test_table_2 [id=618889a5c8d145b6b0d2f74d197b3272]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:54.030102 30488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cfaffe32f00e434faf2580b5fbba1fb5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:54.037355 31305 tablet_bootstrap.cc:492] T 88537bae3b3c401086e601638bc42e43 P c0fad54e965647aaaa34b56674646cb4: Bootstrap starting.
I20240501 14:03:54.038729 31307 tablet_bootstrap.cc:492] T 88537bae3b3c401086e601638bc42e43 P a3e62ccad691413ea2554182f73418a3: Bootstrap starting.
I20240501 14:03:54.039978 31305 tablet_bootstrap.cc:654] T 88537bae3b3c401086e601638bc42e43 P c0fad54e965647aaaa34b56674646cb4: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:54.041817 31307 tablet_bootstrap.cc:654] T 88537bae3b3c401086e601638bc42e43 P a3e62ccad691413ea2554182f73418a3: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:54.047124 31307 tablet_bootstrap.cc:492] T 88537bae3b3c401086e601638bc42e43 P a3e62ccad691413ea2554182f73418a3: No bootstrap required, opened a new log
I20240501 14:03:54.047369 31307 ts_tablet_manager.cc:1367] T 88537bae3b3c401086e601638bc42e43 P a3e62ccad691413ea2554182f73418a3: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240501 14:03:54.047960 31305 tablet_bootstrap.cc:492] T 88537bae3b3c401086e601638bc42e43 P c0fad54e965647aaaa34b56674646cb4: No bootstrap required, opened a new log
I20240501 14:03:54.048118 31305 ts_tablet_manager.cc:1367] T 88537bae3b3c401086e601638bc42e43 P c0fad54e965647aaaa34b56674646cb4: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20240501 14:03:54.048238 31307 raft_consensus.cc:348] T 88537bae3b3c401086e601638bc42e43 P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } }
I20240501 14:03:54.048517 31307 raft_consensus.cc:374] T 88537bae3b3c401086e601638bc42e43 P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:54.048653 31307 raft_consensus.cc:729] T 88537bae3b3c401086e601638bc42e43 P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3e62ccad691413ea2554182f73418a3, State: Initialized, Role: FOLLOWER
I20240501 14:03:54.048872 31305 raft_consensus.cc:348] T 88537bae3b3c401086e601638bc42e43 P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } }
I20240501 14:03:54.049074 31305 raft_consensus.cc:374] T 88537bae3b3c401086e601638bc42e43 P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:54.049170 31305 raft_consensus.cc:729] T 88537bae3b3c401086e601638bc42e43 P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0fad54e965647aaaa34b56674646cb4, State: Initialized, Role: FOLLOWER
I20240501 14:03:54.048969 31307 consensus_queue.cc:260] T 88537bae3b3c401086e601638bc42e43 P a3e62ccad691413ea2554182f73418a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } }
I20240501 14:03:54.049445 31305 consensus_queue.cc:260] T 88537bae3b3c401086e601638bc42e43 P c0fad54e965647aaaa34b56674646cb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } }
I20240501 14:03:54.049924 31307 ts_tablet_manager.cc:1398] T 88537bae3b3c401086e601638bc42e43 P a3e62ccad691413ea2554182f73418a3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:03:54.050489 31303 tablet_bootstrap.cc:492] T 91e42d11fe064d28b457e1bddff845b2 P 99f6ca2d65d94b33b7a1d2341c24179a: Bootstrap starting.
I20240501 14:03:54.052042 31303 tablet_bootstrap.cc:654] T 91e42d11fe064d28b457e1bddff845b2 P 99f6ca2d65d94b33b7a1d2341c24179a: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:54.054611 31307 tablet_bootstrap.cc:492] T 91e42d11fe064d28b457e1bddff845b2 P a3e62ccad691413ea2554182f73418a3: Bootstrap starting.
I20240501 14:03:54.056026 31307 tablet_bootstrap.cc:654] T 91e42d11fe064d28b457e1bddff845b2 P a3e62ccad691413ea2554182f73418a3: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:54.056973 31310 tablet_bootstrap.cc:492] T cfaffe32f00e434faf2580b5fbba1fb5 P 870fbdcfdac84bacae0ad22dba25f626: Bootstrap starting.
I20240501 14:03:54.060016 31303 tablet_bootstrap.cc:492] T 91e42d11fe064d28b457e1bddff845b2 P 99f6ca2d65d94b33b7a1d2341c24179a: No bootstrap required, opened a new log
I20240501 14:03:54.060158 31303 ts_tablet_manager.cc:1367] T 91e42d11fe064d28b457e1bddff845b2 P 99f6ca2d65d94b33b7a1d2341c24179a: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.002s
I20240501 14:03:54.060564 31310 tablet_bootstrap.cc:654] T cfaffe32f00e434faf2580b5fbba1fb5 P 870fbdcfdac84bacae0ad22dba25f626: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:54.060849 31303 raft_consensus.cc:348] T 91e42d11fe064d28b457e1bddff845b2 P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } }
I20240501 14:03:54.061045 31303 raft_consensus.cc:374] T 91e42d11fe064d28b457e1bddff845b2 P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:54.061132 31303 raft_consensus.cc:729] T 91e42d11fe064d28b457e1bddff845b2 P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99f6ca2d65d94b33b7a1d2341c24179a, State: Initialized, Role: FOLLOWER
I20240501 14:03:54.061625 31306 tablet_bootstrap.cc:492] T cfaffe32f00e434faf2580b5fbba1fb5 P 9964c1c76dd14f7dbce766b0c022ab51: Bootstrap starting.
I20240501 14:03:54.062263 31305 ts_tablet_manager.cc:1398] T 88537bae3b3c401086e601638bc42e43 P c0fad54e965647aaaa34b56674646cb4: Time spent starting tablet: real 0.014s user 0.005s sys 0.003s
I20240501 14:03:54.062585 31303 consensus_queue.cc:260] T 91e42d11fe064d28b457e1bddff845b2 P 99f6ca2d65d94b33b7a1d2341c24179a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } }
I20240501 14:03:54.063216 31303 ts_tablet_manager.cc:1398] T 91e42d11fe064d28b457e1bddff845b2 P 99f6ca2d65d94b33b7a1d2341c24179a: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:03:54.064505 31313 tablet_bootstrap.cc:492] T cfaffe32f00e434faf2580b5fbba1fb5 P 8883498098f4454bbfe563fb83d31899: Bootstrap starting.
I20240501 14:03:54.064812 31305 tablet_bootstrap.cc:492] T 91e42d11fe064d28b457e1bddff845b2 P c0fad54e965647aaaa34b56674646cb4: Bootstrap starting.
I20240501 14:03:54.066483 31306 tablet_bootstrap.cc:654] T cfaffe32f00e434faf2580b5fbba1fb5 P 9964c1c76dd14f7dbce766b0c022ab51: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:54.068742 31305 tablet_bootstrap.cc:654] T 91e42d11fe064d28b457e1bddff845b2 P c0fad54e965647aaaa34b56674646cb4: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:54.070595 31306 tablet_bootstrap.cc:492] T cfaffe32f00e434faf2580b5fbba1fb5 P 9964c1c76dd14f7dbce766b0c022ab51: No bootstrap required, opened a new log
I20240501 14:03:54.070776 31306 ts_tablet_manager.cc:1367] T cfaffe32f00e434faf2580b5fbba1fb5 P 9964c1c76dd14f7dbce766b0c022ab51: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.002s
I20240501 14:03:54.071465 31306 raft_consensus.cc:348] T cfaffe32f00e434faf2580b5fbba1fb5 P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:03:54.071869 31310 tablet_bootstrap.cc:492] T cfaffe32f00e434faf2580b5fbba1fb5 P 870fbdcfdac84bacae0ad22dba25f626: No bootstrap required, opened a new log
I20240501 14:03:54.072103 31310 ts_tablet_manager.cc:1367] T cfaffe32f00e434faf2580b5fbba1fb5 P 870fbdcfdac84bacae0ad22dba25f626: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20240501 14:03:54.072373 31306 raft_consensus.cc:374] T cfaffe32f00e434faf2580b5fbba1fb5 P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:54.072636 31306 raft_consensus.cc:729] T cfaffe32f00e434faf2580b5fbba1fb5 P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9964c1c76dd14f7dbce766b0c022ab51, State: Initialized, Role: FOLLOWER
I20240501 14:03:54.072865 31310 raft_consensus.cc:348] T cfaffe32f00e434faf2580b5fbba1fb5 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:03:54.073159 31310 raft_consensus.cc:374] T cfaffe32f00e434faf2580b5fbba1fb5 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:54.073076 31306 consensus_queue.cc:260] T cfaffe32f00e434faf2580b5fbba1fb5 P 9964c1c76dd14f7dbce766b0c022ab51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:03:54.073313 31310 raft_consensus.cc:729] T cfaffe32f00e434faf2580b5fbba1fb5 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 870fbdcfdac84bacae0ad22dba25f626, State: Initialized, Role: FOLLOWER
I20240501 14:03:54.073734 31310 consensus_queue.cc:260] T cfaffe32f00e434faf2580b5fbba1fb5 P 870fbdcfdac84bacae0ad22dba25f626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:03:54.064841 31303 tablet_bootstrap.cc:492] T 88537bae3b3c401086e601638bc42e43 P 99f6ca2d65d94b33b7a1d2341c24179a: Bootstrap starting.
I20240501 14:03:54.075934 31303 tablet_bootstrap.cc:654] T 88537bae3b3c401086e601638bc42e43 P 99f6ca2d65d94b33b7a1d2341c24179a: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:54.077908 31310 ts_tablet_manager.cc:1398] T cfaffe32f00e434faf2580b5fbba1fb5 P 870fbdcfdac84bacae0ad22dba25f626: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240501 14:03:54.078882 31313 tablet_bootstrap.cc:654] T cfaffe32f00e434faf2580b5fbba1fb5 P 8883498098f4454bbfe563fb83d31899: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:54.080607 31306 ts_tablet_manager.cc:1398] T cfaffe32f00e434faf2580b5fbba1fb5 P 9964c1c76dd14f7dbce766b0c022ab51: Time spent starting tablet: real 0.010s user 0.002s sys 0.002s
I20240501 14:03:54.083298 31303 tablet_bootstrap.cc:492] T 88537bae3b3c401086e601638bc42e43 P 99f6ca2d65d94b33b7a1d2341c24179a: No bootstrap required, opened a new log
I20240501 14:03:54.083509 31305 tablet_bootstrap.cc:492] T 91e42d11fe064d28b457e1bddff845b2 P c0fad54e965647aaaa34b56674646cb4: No bootstrap required, opened a new log
I20240501 14:03:54.083509 31307 tablet_bootstrap.cc:492] T 91e42d11fe064d28b457e1bddff845b2 P a3e62ccad691413ea2554182f73418a3: No bootstrap required, opened a new log
I20240501 14:03:54.083550 31303 ts_tablet_manager.cc:1367] T 88537bae3b3c401086e601638bc42e43 P 99f6ca2d65d94b33b7a1d2341c24179a: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.003s
I20240501 14:03:54.083681 31307 ts_tablet_manager.cc:1367] T 91e42d11fe064d28b457e1bddff845b2 P a3e62ccad691413ea2554182f73418a3: Time spent bootstrapping tablet: real 0.029s user 0.005s sys 0.011s
I20240501 14:03:54.083772 31305 ts_tablet_manager.cc:1367] T 91e42d11fe064d28b457e1bddff845b2 P c0fad54e965647aaaa34b56674646cb4: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.011s
I20240501 14:03:54.084471 31307 raft_consensus.cc:348] T 91e42d11fe064d28b457e1bddff845b2 P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } }
I20240501 14:03:54.084471 31305 raft_consensus.cc:348] T 91e42d11fe064d28b457e1bddff845b2 P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } }
I20240501 14:03:54.084688 31307 raft_consensus.cc:374] T 91e42d11fe064d28b457e1bddff845b2 P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:54.084769 31307 raft_consensus.cc:729] T 91e42d11fe064d28b457e1bddff845b2 P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3e62ccad691413ea2554182f73418a3, State: Initialized, Role: FOLLOWER
I20240501 14:03:54.084851 31305 raft_consensus.cc:374] T 91e42d11fe064d28b457e1bddff845b2 P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:54.084594 31303 raft_consensus.cc:348] T 88537bae3b3c401086e601638bc42e43 P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } }
I20240501 14:03:54.084935 31305 raft_consensus.cc:729] T 91e42d11fe064d28b457e1bddff845b2 P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0fad54e965647aaaa34b56674646cb4, State: Initialized, Role: FOLLOWER
I20240501 14:03:54.085006 31307 consensus_queue.cc:260] T 91e42d11fe064d28b457e1bddff845b2 P a3e62ccad691413ea2554182f73418a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } }
I20240501 14:03:54.085142 31303 raft_consensus.cc:374] T 88537bae3b3c401086e601638bc42e43 P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:54.085171 31305 consensus_queue.cc:260] T 91e42d11fe064d28b457e1bddff845b2 P c0fad54e965647aaaa34b56674646cb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } }
I20240501 14:03:54.085355 31303 raft_consensus.cc:729] T 88537bae3b3c401086e601638bc42e43 P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99f6ca2d65d94b33b7a1d2341c24179a, State: Initialized, Role: FOLLOWER
I20240501 14:03:54.085532 31307 ts_tablet_manager.cc:1398] T 91e42d11fe064d28b457e1bddff845b2 P a3e62ccad691413ea2554182f73418a3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:03:54.085724 31303 consensus_queue.cc:260] T 88537bae3b3c401086e601638bc42e43 P 99f6ca2d65d94b33b7a1d2341c24179a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } }
I20240501 14:03:54.086256 31303 ts_tablet_manager.cc:1398] T 88537bae3b3c401086e601638bc42e43 P 99f6ca2d65d94b33b7a1d2341c24179a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:03:54.086324 31305 ts_tablet_manager.cc:1398] T 91e42d11fe064d28b457e1bddff845b2 P c0fad54e965647aaaa34b56674646cb4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:03:54.087458 31313 tablet_bootstrap.cc:492] T cfaffe32f00e434faf2580b5fbba1fb5 P 8883498098f4454bbfe563fb83d31899: No bootstrap required, opened a new log
I20240501 14:03:54.088146 31313 ts_tablet_manager.cc:1367] T cfaffe32f00e434faf2580b5fbba1fb5 P 8883498098f4454bbfe563fb83d31899: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.003s
I20240501 14:03:54.088907 31313 raft_consensus.cc:348] T cfaffe32f00e434faf2580b5fbba1fb5 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:03:54.089143 31313 raft_consensus.cc:374] T cfaffe32f00e434faf2580b5fbba1fb5 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:54.089279 31313 raft_consensus.cc:729] T cfaffe32f00e434faf2580b5fbba1fb5 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8883498098f4454bbfe563fb83d31899, State: Initialized, Role: FOLLOWER
I20240501 14:03:54.089517 31313 consensus_queue.cc:260] T cfaffe32f00e434faf2580b5fbba1fb5 P 8883498098f4454bbfe563fb83d31899 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:03:54.090199 31313 ts_tablet_manager.cc:1398] T cfaffe32f00e434faf2580b5fbba1fb5 P 8883498098f4454bbfe563fb83d31899: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:03:54.090677 31232 raft_consensus.cc:482] T cfaffe32f00e434faf2580b5fbba1fb5 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:03:54.090852 31232 raft_consensus.cc:504] T cfaffe32f00e434faf2580b5fbba1fb5 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:03:54.091460 31232 leader_election.cc:288] T cfaffe32f00e434faf2580b5fbba1fb5 P 8883498098f4454bbfe563fb83d31899 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 870fbdcfdac84bacae0ad22dba25f626 (127.29.124.132:44267), 9964c1c76dd14f7dbce766b0c022ab51 (127.29.124.134:38101)
I20240501 14:03:54.091977 30771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfaffe32f00e434faf2580b5fbba1fb5" candidate_uuid: "8883498098f4454bbfe563fb83d31899" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "870fbdcfdac84bacae0ad22dba25f626" is_pre_election: true
I20240501 14:03:54.091960 31033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfaffe32f00e434faf2580b5fbba1fb5" candidate_uuid: "8883498098f4454bbfe563fb83d31899" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9964c1c76dd14f7dbce766b0c022ab51" is_pre_election: true
I20240501 14:03:54.092165 30771 raft_consensus.cc:2454] T cfaffe32f00e434faf2580b5fbba1fb5 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8883498098f4454bbfe563fb83d31899 in term 0.
I20240501 14:03:54.092185 31033 raft_consensus.cc:2454] T cfaffe32f00e434faf2580b5fbba1fb5 P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8883498098f4454bbfe563fb83d31899 in term 0.
I20240501 14:03:54.092573 30445 leader_election.cc:302] T cfaffe32f00e434faf2580b5fbba1fb5 P 8883498098f4454bbfe563fb83d31899 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 870fbdcfdac84bacae0ad22dba25f626, 8883498098f4454bbfe563fb83d31899; no voters:
I20240501 14:03:54.092839 31232 raft_consensus.cc:2781] T cfaffe32f00e434faf2580b5fbba1fb5 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:03:54.092926 31232 raft_consensus.cc:482] T cfaffe32f00e434faf2580b5fbba1fb5 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:03:54.093021 31232 raft_consensus.cc:3037] T cfaffe32f00e434faf2580b5fbba1fb5 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:54.094813 31232 raft_consensus.cc:504] T cfaffe32f00e434faf2580b5fbba1fb5 P 8883498098f4454bbfe563fb83d31899 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:03:54.095345 31232 leader_election.cc:288] T cfaffe32f00e434faf2580b5fbba1fb5 P 8883498098f4454bbfe563fb83d31899 [CANDIDATE]: Term 1 election: Requested vote from peers 870fbdcfdac84bacae0ad22dba25f626 (127.29.124.132:44267), 9964c1c76dd14f7dbce766b0c022ab51 (127.29.124.134:38101)
I20240501 14:03:54.095633 30771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfaffe32f00e434faf2580b5fbba1fb5" candidate_uuid: "8883498098f4454bbfe563fb83d31899" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "870fbdcfdac84bacae0ad22dba25f626"
I20240501 14:03:54.095814 30771 raft_consensus.cc:3037] T cfaffe32f00e434faf2580b5fbba1fb5 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:54.095939 31033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cfaffe32f00e434faf2580b5fbba1fb5" candidate_uuid: "8883498098f4454bbfe563fb83d31899" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9964c1c76dd14f7dbce766b0c022ab51"
I20240501 14:03:54.096114 31033 raft_consensus.cc:3037] T cfaffe32f00e434faf2580b5fbba1fb5 P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:54.097715 30771 raft_consensus.cc:2454] T cfaffe32f00e434faf2580b5fbba1fb5 P 870fbdcfdac84bacae0ad22dba25f626 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8883498098f4454bbfe563fb83d31899 in term 1.
I20240501 14:03:54.098085 30445 leader_election.cc:302] T cfaffe32f00e434faf2580b5fbba1fb5 P 8883498098f4454bbfe563fb83d31899 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 870fbdcfdac84bacae0ad22dba25f626, 8883498098f4454bbfe563fb83d31899; no voters:
I20240501 14:03:54.098361 31232 raft_consensus.cc:2781] T cfaffe32f00e434faf2580b5fbba1fb5 P 8883498098f4454bbfe563fb83d31899 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:03:54.098376 31033 raft_consensus.cc:2454] T cfaffe32f00e434faf2580b5fbba1fb5 P 9964c1c76dd14f7dbce766b0c022ab51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8883498098f4454bbfe563fb83d31899 in term 1.
I20240501 14:03:54.101451 31232 raft_consensus.cc:686] T cfaffe32f00e434faf2580b5fbba1fb5 P 8883498098f4454bbfe563fb83d31899 [term 1 LEADER]: Becoming Leader. State: Replica: 8883498098f4454bbfe563fb83d31899, State: Running, Role: LEADER
I20240501 14:03:54.101680 31232 consensus_queue.cc:237] T cfaffe32f00e434faf2580b5fbba1fb5 P 8883498098f4454bbfe563fb83d31899 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:03:54.103749 30229 catalog_manager.cc:5484] T cfaffe32f00e434faf2580b5fbba1fb5 P 8883498098f4454bbfe563fb83d31899 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8883498098f4454bbfe563fb83d31899 (127.29.124.130). New cstate: current_term: 1 leader_uuid: "8883498098f4454bbfe563fb83d31899" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } health_report { overall_health: HEALTHY } } }
I20240501 14:03:54.124879 31311 raft_consensus.cc:482] T 88537bae3b3c401086e601638bc42e43 P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:03:54.125065 31311 raft_consensus.cc:504] T 88537bae3b3c401086e601638bc42e43 P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } }
I20240501 14:03:54.126565 31311 leader_election.cc:288] T 88537bae3b3c401086e601638bc42e43 P c0fad54e965647aaaa34b56674646cb4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3e62ccad691413ea2554182f73418a3 (127.29.124.131:43965), 99f6ca2d65d94b33b7a1d2341c24179a (127.29.124.133:35195)
I20240501 14:03:54.147539 30901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88537bae3b3c401086e601638bc42e43" candidate_uuid: "c0fad54e965647aaaa34b56674646cb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" is_pre_election: true
I20240501 14:03:54.147812 30901 raft_consensus.cc:2454] T 88537bae3b3c401086e601638bc42e43 P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0fad54e965647aaaa34b56674646cb4 in term 0.
I20240501 14:03:54.147878 30638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88537bae3b3c401086e601638bc42e43" candidate_uuid: "c0fad54e965647aaaa34b56674646cb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3e62ccad691413ea2554182f73418a3" is_pre_election: true
I20240501 14:03:54.148125 30638 raft_consensus.cc:2454] T 88537bae3b3c401086e601638bc42e43 P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0fad54e965647aaaa34b56674646cb4 in term 0.
I20240501 14:03:54.148339 30313 leader_election.cc:302] T 88537bae3b3c401086e601638bc42e43 P c0fad54e965647aaaa34b56674646cb4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99f6ca2d65d94b33b7a1d2341c24179a, c0fad54e965647aaaa34b56674646cb4; no voters:
I20240501 14:03:54.148763 31311 raft_consensus.cc:2781] T 88537bae3b3c401086e601638bc42e43 P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:03:54.148867 31311 raft_consensus.cc:482] T 88537bae3b3c401086e601638bc42e43 P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:03:54.148949 31311 raft_consensus.cc:3037] T 88537bae3b3c401086e601638bc42e43 P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:54.150954 31311 raft_consensus.cc:504] T 88537bae3b3c401086e601638bc42e43 P c0fad54e965647aaaa34b56674646cb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } }
I20240501 14:03:54.151575 31311 leader_election.cc:288] T 88537bae3b3c401086e601638bc42e43 P c0fad54e965647aaaa34b56674646cb4 [CANDIDATE]: Term 1 election: Requested vote from peers a3e62ccad691413ea2554182f73418a3 (127.29.124.131:43965), 99f6ca2d65d94b33b7a1d2341c24179a (127.29.124.133:35195)
I20240501 14:03:54.151859 30638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88537bae3b3c401086e601638bc42e43" candidate_uuid: "c0fad54e965647aaaa34b56674646cb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3e62ccad691413ea2554182f73418a3"
I20240501 14:03:54.152063 30638 raft_consensus.cc:3037] T 88537bae3b3c401086e601638bc42e43 P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:54.153869 30638 raft_consensus.cc:2454] T 88537bae3b3c401086e601638bc42e43 P a3e62ccad691413ea2554182f73418a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0fad54e965647aaaa34b56674646cb4 in term 1.
I20240501 14:03:54.154243 30311 leader_election.cc:302] T 88537bae3b3c401086e601638bc42e43 P c0fad54e965647aaaa34b56674646cb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3e62ccad691413ea2554182f73418a3, c0fad54e965647aaaa34b56674646cb4; no voters:
I20240501 14:03:54.154211 30901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88537bae3b3c401086e601638bc42e43" candidate_uuid: "c0fad54e965647aaaa34b56674646cb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99f6ca2d65d94b33b7a1d2341c24179a"
I20240501 14:03:54.154417 30901 raft_consensus.cc:3037] T 88537bae3b3c401086e601638bc42e43 P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:54.154481 31311 raft_consensus.cc:2781] T 88537bae3b3c401086e601638bc42e43 P c0fad54e965647aaaa34b56674646cb4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:03:54.154909 31311 raft_consensus.cc:686] T 88537bae3b3c401086e601638bc42e43 P c0fad54e965647aaaa34b56674646cb4 [term 1 LEADER]: Becoming Leader. State: Replica: c0fad54e965647aaaa34b56674646cb4, State: Running, Role: LEADER
I20240501 14:03:54.155452 31311 consensus_queue.cc:237] T 88537bae3b3c401086e601638bc42e43 P c0fad54e965647aaaa34b56674646cb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } }
I20240501 14:03:54.157163 30901 raft_consensus.cc:2454] T 88537bae3b3c401086e601638bc42e43 P 99f6ca2d65d94b33b7a1d2341c24179a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0fad54e965647aaaa34b56674646cb4 in term 1.
I20240501 14:03:54.159294 30229 catalog_manager.cc:5484] T 88537bae3b3c401086e601638bc42e43 P c0fad54e965647aaaa34b56674646cb4 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0fad54e965647aaaa34b56674646cb4 (127.29.124.129). New cstate: current_term: 1 leader_uuid: "c0fad54e965647aaaa34b56674646cb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } health_report { overall_health: UNKNOWN } } }
I20240501 14:03:54.218956 31304 raft_consensus.cc:482] T 91e42d11fe064d28b457e1bddff845b2 P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:03:54.219121 31304 raft_consensus.cc:504] T 91e42d11fe064d28b457e1bddff845b2 P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } }
I20240501 14:03:54.219837 31304 leader_election.cc:288] T 91e42d11fe064d28b457e1bddff845b2 P 99f6ca2d65d94b33b7a1d2341c24179a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3e62ccad691413ea2554182f73418a3 (127.29.124.131:43965), c0fad54e965647aaaa34b56674646cb4 (127.29.124.129:42363)
I20240501 14:03:54.220325 30638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e42d11fe064d28b457e1bddff845b2" candidate_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3e62ccad691413ea2554182f73418a3" is_pre_election: true
I20240501 14:03:54.220417 30378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e42d11fe064d28b457e1bddff845b2" candidate_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0fad54e965647aaaa34b56674646cb4" is_pre_election: true
I20240501 14:03:54.220551 30638 raft_consensus.cc:2454] T 91e42d11fe064d28b457e1bddff845b2 P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99f6ca2d65d94b33b7a1d2341c24179a in term 0.
I20240501 14:03:54.220650 30378 raft_consensus.cc:2454] T 91e42d11fe064d28b457e1bddff845b2 P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99f6ca2d65d94b33b7a1d2341c24179a in term 0.
I20240501 14:03:54.221146 30835 leader_election.cc:302] T 91e42d11fe064d28b457e1bddff845b2 P 99f6ca2d65d94b33b7a1d2341c24179a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99f6ca2d65d94b33b7a1d2341c24179a, a3e62ccad691413ea2554182f73418a3; no voters:
I20240501 14:03:54.221454 31304 raft_consensus.cc:2781] T 91e42d11fe064d28b457e1bddff845b2 P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:03:54.221549 31304 raft_consensus.cc:482] T 91e42d11fe064d28b457e1bddff845b2 P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:03:54.221632 31304 raft_consensus.cc:3037] T 91e42d11fe064d28b457e1bddff845b2 P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:54.223520 31304 raft_consensus.cc:504] T 91e42d11fe064d28b457e1bddff845b2 P 99f6ca2d65d94b33b7a1d2341c24179a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } }
I20240501 14:03:54.224009 31304 leader_election.cc:288] T 91e42d11fe064d28b457e1bddff845b2 P 99f6ca2d65d94b33b7a1d2341c24179a [CANDIDATE]: Term 1 election: Requested vote from peers a3e62ccad691413ea2554182f73418a3 (127.29.124.131:43965), c0fad54e965647aaaa34b56674646cb4 (127.29.124.129:42363)
I20240501 14:03:54.224442 30638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e42d11fe064d28b457e1bddff845b2" candidate_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3e62ccad691413ea2554182f73418a3"
I20240501 14:03:54.224552 30378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91e42d11fe064d28b457e1bddff845b2" candidate_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0fad54e965647aaaa34b56674646cb4"
I20240501 14:03:54.224649 30638 raft_consensus.cc:3037] T 91e42d11fe064d28b457e1bddff845b2 P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:54.224759 30378 raft_consensus.cc:3037] T 91e42d11fe064d28b457e1bddff845b2 P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:54.227407 30378 raft_consensus.cc:2454] T 91e42d11fe064d28b457e1bddff845b2 P c0fad54e965647aaaa34b56674646cb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99f6ca2d65d94b33b7a1d2341c24179a in term 1.
I20240501 14:03:54.227407 30638 raft_consensus.cc:2454] T 91e42d11fe064d28b457e1bddff845b2 P a3e62ccad691413ea2554182f73418a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99f6ca2d65d94b33b7a1d2341c24179a in term 1.
I20240501 14:03:54.227792 30836 leader_election.cc:302] T 91e42d11fe064d28b457e1bddff845b2 P 99f6ca2d65d94b33b7a1d2341c24179a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99f6ca2d65d94b33b7a1d2341c24179a, c0fad54e965647aaaa34b56674646cb4; no voters:
I20240501 14:03:54.228055 31304 raft_consensus.cc:2781] T 91e42d11fe064d28b457e1bddff845b2 P 99f6ca2d65d94b33b7a1d2341c24179a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:03:54.228183 31304 raft_consensus.cc:686] T 91e42d11fe064d28b457e1bddff845b2 P 99f6ca2d65d94b33b7a1d2341c24179a [term 1 LEADER]: Becoming Leader. State: Replica: 99f6ca2d65d94b33b7a1d2341c24179a, State: Running, Role: LEADER
I20240501 14:03:54.228471 31304 consensus_queue.cc:237] T 91e42d11fe064d28b457e1bddff845b2 P 99f6ca2d65d94b33b7a1d2341c24179a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } }
I20240501 14:03:54.231016 30229 catalog_manager.cc:5484] T 91e42d11fe064d28b457e1bddff845b2 P 99f6ca2d65d94b33b7a1d2341c24179a reported cstate change: term changed from 0 to 1, leader changed from <none> to 99f6ca2d65d94b33b7a1d2341c24179a (127.29.124.133). New cstate: current_term: 1 leader_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } health_report { overall_health: UNKNOWN } } }
I20240501 14:03:54.561532 31311 consensus_queue.cc:1035] T 88537bae3b3c401086e601638bc42e43 P c0fad54e965647aaaa34b56674646cb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:54.568495 31311 consensus_queue.cc:1035] T 88537bae3b3c401086e601638bc42e43 P c0fad54e965647aaaa34b56674646cb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:54.572228 31232 consensus_queue.cc:1035] T cfaffe32f00e434faf2580b5fbba1fb5 P 8883498098f4454bbfe563fb83d31899 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:54.578418 31321 consensus_queue.cc:1035] T cfaffe32f00e434faf2580b5fbba1fb5 P 8883498098f4454bbfe563fb83d31899 [LEADER]: Connected to new peer: Peer: permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:03:54.594364 31330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:03:54.594647 31330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:03:54.601827 31330 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240501 14:03:54.698791 31304 consensus_queue.cc:1035] T 91e42d11fe064d28b457e1bddff845b2 P 99f6ca2d65d94b33b7a1d2341c24179a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:54.707520 31304 consensus_queue.cc:1035] T 91e42d11fe064d28b457e1bddff845b2 P 99f6ca2d65d94b33b7a1d2341c24179a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 259.236 ms
time per row: 0.129618 ms
I20240501 14:03:55.005733 30235 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35960:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:03:55.015272 31012 tablet_service.cc:1467] Processing CreateTablet for tablet a0465b32deb54a5cb18fdd132d33285d (DEFAULT_TABLE table=rebalance_test_table_3 [id=50cdb5f3237b410e8b722eebe5a520a1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:55.015743 31012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0465b32deb54a5cb18fdd132d33285d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:55.015719 30488 tablet_service.cc:1467] Processing CreateTablet for tablet a0465b32deb54a5cb18fdd132d33285d (DEFAULT_TABLE table=rebalance_test_table_3 [id=50cdb5f3237b410e8b722eebe5a520a1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:55.015964 30750 tablet_service.cc:1467] Processing CreateTablet for tablet a0465b32deb54a5cb18fdd132d33285d (DEFAULT_TABLE table=rebalance_test_table_3 [id=50cdb5f3237b410e8b722eebe5a520a1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:55.016144 30488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0465b32deb54a5cb18fdd132d33285d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:55.016366 30750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0465b32deb54a5cb18fdd132d33285d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:55.016594 30881 tablet_service.cc:1467] Processing CreateTablet for tablet dcd50830d6fd4aa3bdcc89c4e60c13ca (DEFAULT_TABLE table=rebalance_test_table_3 [id=50cdb5f3237b410e8b722eebe5a520a1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:55.016981 30881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcd50830d6fd4aa3bdcc89c4e60c13ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:55.021026 30751 tablet_service.cc:1467] Processing CreateTablet for tablet 04e140965a314fbca43595aed44942b4 (DEFAULT_TABLE table=rebalance_test_table_3 [id=50cdb5f3237b410e8b722eebe5a520a1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:55.021067 30619 tablet_service.cc:1467] Processing CreateTablet for tablet dcd50830d6fd4aa3bdcc89c4e60c13ca (DEFAULT_TABLE table=rebalance_test_table_3 [id=50cdb5f3237b410e8b722eebe5a520a1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:55.021400 30751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04e140965a314fbca43595aed44942b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:55.021899 31013 tablet_service.cc:1467] Processing CreateTablet for tablet 04e140965a314fbca43595aed44942b4 (DEFAULT_TABLE table=rebalance_test_table_3 [id=50cdb5f3237b410e8b722eebe5a520a1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:55.022245 31013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04e140965a314fbca43595aed44942b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:55.023046 30358 tablet_service.cc:1467] Processing CreateTablet for tablet dcd50830d6fd4aa3bdcc89c4e60c13ca (DEFAULT_TABLE table=rebalance_test_table_3 [id=50cdb5f3237b410e8b722eebe5a520a1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:55.023434 30358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcd50830d6fd4aa3bdcc89c4e60c13ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:55.025661 30619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dcd50830d6fd4aa3bdcc89c4e60c13ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:55.029457 30489 tablet_service.cc:1467] Processing CreateTablet for tablet 04e140965a314fbca43595aed44942b4 (DEFAULT_TABLE table=rebalance_test_table_3 [id=50cdb5f3237b410e8b722eebe5a520a1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:55.029847 30489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04e140965a314fbca43595aed44942b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:55.032763 31395 tablet_bootstrap.cc:492] T 04e140965a314fbca43595aed44942b4 P 9964c1c76dd14f7dbce766b0c022ab51: Bootstrap starting.
I20240501 14:03:55.034602 31395 tablet_bootstrap.cc:654] T 04e140965a314fbca43595aed44942b4 P 9964c1c76dd14f7dbce766b0c022ab51: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:55.038904 31397 tablet_bootstrap.cc:492] T dcd50830d6fd4aa3bdcc89c4e60c13ca P 99f6ca2d65d94b33b7a1d2341c24179a: Bootstrap starting.
I20240501 14:03:55.039573 31396 tablet_bootstrap.cc:492] T dcd50830d6fd4aa3bdcc89c4e60c13ca P a3e62ccad691413ea2554182f73418a3: Bootstrap starting.
I20240501 14:03:55.041326 31396 tablet_bootstrap.cc:654] T dcd50830d6fd4aa3bdcc89c4e60c13ca P a3e62ccad691413ea2554182f73418a3: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:55.041989 31394 tablet_bootstrap.cc:492] T 04e140965a314fbca43595aed44942b4 P 870fbdcfdac84bacae0ad22dba25f626: Bootstrap starting.
I20240501 14:03:55.043187 31392 tablet_bootstrap.cc:492] T a0465b32deb54a5cb18fdd132d33285d P 8883498098f4454bbfe563fb83d31899: Bootstrap starting.
I20240501 14:03:55.043553 31393 tablet_bootstrap.cc:492] T dcd50830d6fd4aa3bdcc89c4e60c13ca P c0fad54e965647aaaa34b56674646cb4: Bootstrap starting.
I20240501 14:03:55.043924 31394 tablet_bootstrap.cc:654] T 04e140965a314fbca43595aed44942b4 P 870fbdcfdac84bacae0ad22dba25f626: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:55.044647 31397 tablet_bootstrap.cc:654] T dcd50830d6fd4aa3bdcc89c4e60c13ca P 99f6ca2d65d94b33b7a1d2341c24179a: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:55.044847 31392 tablet_bootstrap.cc:654] T a0465b32deb54a5cb18fdd132d33285d P 8883498098f4454bbfe563fb83d31899: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:55.045820 31393 tablet_bootstrap.cc:654] T dcd50830d6fd4aa3bdcc89c4e60c13ca P c0fad54e965647aaaa34b56674646cb4: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:55.047075 31395 tablet_bootstrap.cc:492] T 04e140965a314fbca43595aed44942b4 P 9964c1c76dd14f7dbce766b0c022ab51: No bootstrap required, opened a new log
I20240501 14:03:55.047219 31395 ts_tablet_manager.cc:1367] T 04e140965a314fbca43595aed44942b4 P 9964c1c76dd14f7dbce766b0c022ab51: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.002s
I20240501 14:03:55.047358 31396 tablet_bootstrap.cc:492] T dcd50830d6fd4aa3bdcc89c4e60c13ca P a3e62ccad691413ea2554182f73418a3: No bootstrap required, opened a new log
I20240501 14:03:55.047489 31396 ts_tablet_manager.cc:1367] T dcd50830d6fd4aa3bdcc89c4e60c13ca P a3e62ccad691413ea2554182f73418a3: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240501 14:03:55.047953 31395 raft_consensus.cc:348] T 04e140965a314fbca43595aed44942b4 P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:03:55.048270 31395 raft_consensus.cc:374] T 04e140965a314fbca43595aed44942b4 P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:55.048250 31396 raft_consensus.cc:348] T dcd50830d6fd4aa3bdcc89c4e60c13ca P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } }
I20240501 14:03:55.048431 31395 raft_consensus.cc:729] T 04e140965a314fbca43595aed44942b4 P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9964c1c76dd14f7dbce766b0c022ab51, State: Initialized, Role: FOLLOWER
I20240501 14:03:55.048555 31396 raft_consensus.cc:374] T dcd50830d6fd4aa3bdcc89c4e60c13ca P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:55.048691 31396 raft_consensus.cc:729] T dcd50830d6fd4aa3bdcc89c4e60c13ca P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3e62ccad691413ea2554182f73418a3, State: Initialized, Role: FOLLOWER
I20240501 14:03:55.048696 31395 consensus_queue.cc:260] T 04e140965a314fbca43595aed44942b4 P 9964c1c76dd14f7dbce766b0c022ab51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:03:55.049057 31396 consensus_queue.cc:260] T dcd50830d6fd4aa3bdcc89c4e60c13ca P a3e62ccad691413ea2554182f73418a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } }
I20240501 14:03:55.050019 31395 ts_tablet_manager.cc:1398] T 04e140965a314fbca43595aed44942b4 P 9964c1c76dd14f7dbce766b0c022ab51: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:03:55.050833 31395 tablet_bootstrap.cc:492] T a0465b32deb54a5cb18fdd132d33285d P 9964c1c76dd14f7dbce766b0c022ab51: Bootstrap starting.
I20240501 14:03:55.051223 31397 tablet_bootstrap.cc:492] T dcd50830d6fd4aa3bdcc89c4e60c13ca P 99f6ca2d65d94b33b7a1d2341c24179a: No bootstrap required, opened a new log
I20240501 14:03:55.051414 31397 ts_tablet_manager.cc:1367] T dcd50830d6fd4aa3bdcc89c4e60c13ca P 99f6ca2d65d94b33b7a1d2341c24179a: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.003s
I20240501 14:03:55.052361 31395 tablet_bootstrap.cc:654] T a0465b32deb54a5cb18fdd132d33285d P 9964c1c76dd14f7dbce766b0c022ab51: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:55.052253 31397 raft_consensus.cc:348] T dcd50830d6fd4aa3bdcc89c4e60c13ca P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } }
I20240501 14:03:55.052438 31397 raft_consensus.cc:374] T dcd50830d6fd4aa3bdcc89c4e60c13ca P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:55.052558 31397 raft_consensus.cc:729] T dcd50830d6fd4aa3bdcc89c4e60c13ca P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99f6ca2d65d94b33b7a1d2341c24179a, State: Initialized, Role: FOLLOWER
I20240501 14:03:55.052772 31393 tablet_bootstrap.cc:492] T dcd50830d6fd4aa3bdcc89c4e60c13ca P c0fad54e965647aaaa34b56674646cb4: No bootstrap required, opened a new log
I20240501 14:03:55.052970 31393 ts_tablet_manager.cc:1367] T dcd50830d6fd4aa3bdcc89c4e60c13ca P c0fad54e965647aaaa34b56674646cb4: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20240501 14:03:55.052846 31397 consensus_queue.cc:260] T dcd50830d6fd4aa3bdcc89c4e60c13ca P 99f6ca2d65d94b33b7a1d2341c24179a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } }
I20240501 14:03:55.053450 31396 ts_tablet_manager.cc:1398] T dcd50830d6fd4aa3bdcc89c4e60c13ca P a3e62ccad691413ea2554182f73418a3: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:03:55.054006 31397 ts_tablet_manager.cc:1398] T dcd50830d6fd4aa3bdcc89c4e60c13ca P 99f6ca2d65d94b33b7a1d2341c24179a: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:03:55.056833 31394 tablet_bootstrap.cc:492] T 04e140965a314fbca43595aed44942b4 P 870fbdcfdac84bacae0ad22dba25f626: No bootstrap required, opened a new log
I20240501 14:03:55.056978 31393 raft_consensus.cc:348] T dcd50830d6fd4aa3bdcc89c4e60c13ca P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } }
I20240501 14:03:55.057173 31393 raft_consensus.cc:374] T dcd50830d6fd4aa3bdcc89c4e60c13ca P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:55.057228 31394 ts_tablet_manager.cc:1367] T 04e140965a314fbca43595aed44942b4 P 870fbdcfdac84bacae0ad22dba25f626: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20240501 14:03:55.057294 31393 raft_consensus.cc:729] T dcd50830d6fd4aa3bdcc89c4e60c13ca P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0fad54e965647aaaa34b56674646cb4, State: Initialized, Role: FOLLOWER
I20240501 14:03:55.057600 31393 consensus_queue.cc:260] T dcd50830d6fd4aa3bdcc89c4e60c13ca P c0fad54e965647aaaa34b56674646cb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } }
I20240501 14:03:55.058043 31392 tablet_bootstrap.cc:492] T a0465b32deb54a5cb18fdd132d33285d P 8883498098f4454bbfe563fb83d31899: No bootstrap required, opened a new log
I20240501 14:03:55.058180 31392 ts_tablet_manager.cc:1367] T a0465b32deb54a5cb18fdd132d33285d P 8883498098f4454bbfe563fb83d31899: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20240501 14:03:55.058313 31393 ts_tablet_manager.cc:1398] T dcd50830d6fd4aa3bdcc89c4e60c13ca P c0fad54e965647aaaa34b56674646cb4: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240501 14:03:55.058715 31394 raft_consensus.cc:348] T 04e140965a314fbca43595aed44942b4 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:03:55.058895 31394 raft_consensus.cc:374] T 04e140965a314fbca43595aed44942b4 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:55.058991 31394 raft_consensus.cc:729] T 04e140965a314fbca43595aed44942b4 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 870fbdcfdac84bacae0ad22dba25f626, State: Initialized, Role: FOLLOWER
I20240501 14:03:55.058894 31392 raft_consensus.cc:348] T a0465b32deb54a5cb18fdd132d33285d P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:03:55.059089 31392 raft_consensus.cc:374] T a0465b32deb54a5cb18fdd132d33285d P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:55.059176 31392 raft_consensus.cc:729] T a0465b32deb54a5cb18fdd132d33285d P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8883498098f4454bbfe563fb83d31899, State: Initialized, Role: FOLLOWER
I20240501 14:03:55.059185 31394 consensus_queue.cc:260] T 04e140965a314fbca43595aed44942b4 P 870fbdcfdac84bacae0ad22dba25f626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:03:55.059381 31392 consensus_queue.cc:260] T a0465b32deb54a5cb18fdd132d33285d P 8883498098f4454bbfe563fb83d31899 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:03:55.059921 31392 ts_tablet_manager.cc:1398] T a0465b32deb54a5cb18fdd132d33285d P 8883498098f4454bbfe563fb83d31899: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:03:55.060138 31394 ts_tablet_manager.cc:1398] T 04e140965a314fbca43595aed44942b4 P 870fbdcfdac84bacae0ad22dba25f626: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:03:55.060195 31392 tablet_bootstrap.cc:492] T 04e140965a314fbca43595aed44942b4 P 8883498098f4454bbfe563fb83d31899: Bootstrap starting.
I20240501 14:03:55.060379 31394 tablet_bootstrap.cc:492] T a0465b32deb54a5cb18fdd132d33285d P 870fbdcfdac84bacae0ad22dba25f626: Bootstrap starting.
I20240501 14:03:55.061853 31394 tablet_bootstrap.cc:654] T a0465b32deb54a5cb18fdd132d33285d P 870fbdcfdac84bacae0ad22dba25f626: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:55.065800 31392 tablet_bootstrap.cc:654] T 04e140965a314fbca43595aed44942b4 P 8883498098f4454bbfe563fb83d31899: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:55.067402 31392 tablet_bootstrap.cc:492] T 04e140965a314fbca43595aed44942b4 P 8883498098f4454bbfe563fb83d31899: No bootstrap required, opened a new log
I20240501 14:03:55.067548 31392 ts_tablet_manager.cc:1367] T 04e140965a314fbca43595aed44942b4 P 8883498098f4454bbfe563fb83d31899: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:03:55.068087 31392 raft_consensus.cc:348] T 04e140965a314fbca43595aed44942b4 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:03:55.068248 31392 raft_consensus.cc:374] T 04e140965a314fbca43595aed44942b4 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:55.068320 31392 raft_consensus.cc:729] T 04e140965a314fbca43595aed44942b4 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8883498098f4454bbfe563fb83d31899, State: Initialized, Role: FOLLOWER
I20240501 14:03:55.068554 31392 consensus_queue.cc:260] T 04e140965a314fbca43595aed44942b4 P 8883498098f4454bbfe563fb83d31899 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:03:55.069674 31392 ts_tablet_manager.cc:1398] T 04e140965a314fbca43595aed44942b4 P 8883498098f4454bbfe563fb83d31899: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:03:55.070432 31395 tablet_bootstrap.cc:492] T a0465b32deb54a5cb18fdd132d33285d P 9964c1c76dd14f7dbce766b0c022ab51: No bootstrap required, opened a new log
I20240501 14:03:55.070588 31395 ts_tablet_manager.cc:1367] T a0465b32deb54a5cb18fdd132d33285d P 9964c1c76dd14f7dbce766b0c022ab51: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.002s
I20240501 14:03:55.071317 31395 raft_consensus.cc:348] T a0465b32deb54a5cb18fdd132d33285d P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:03:55.071511 31395 raft_consensus.cc:374] T a0465b32deb54a5cb18fdd132d33285d P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:55.071573 31395 raft_consensus.cc:729] T a0465b32deb54a5cb18fdd132d33285d P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9964c1c76dd14f7dbce766b0c022ab51, State: Initialized, Role: FOLLOWER
I20240501 14:03:55.071777 31395 consensus_queue.cc:260] T a0465b32deb54a5cb18fdd132d33285d P 9964c1c76dd14f7dbce766b0c022ab51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:03:55.072216 31395 ts_tablet_manager.cc:1398] T a0465b32deb54a5cb18fdd132d33285d P 9964c1c76dd14f7dbce766b0c022ab51: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:03:55.078532 31394 tablet_bootstrap.cc:492] T a0465b32deb54a5cb18fdd132d33285d P 870fbdcfdac84bacae0ad22dba25f626: No bootstrap required, opened a new log
I20240501 14:03:55.078752 31394 ts_tablet_manager.cc:1367] T a0465b32deb54a5cb18fdd132d33285d P 870fbdcfdac84bacae0ad22dba25f626: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.005s
I20240501 14:03:55.079423 31394 raft_consensus.cc:348] T a0465b32deb54a5cb18fdd132d33285d P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:03:55.079598 31394 raft_consensus.cc:374] T a0465b32deb54a5cb18fdd132d33285d P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:55.079671 31394 raft_consensus.cc:729] T a0465b32deb54a5cb18fdd132d33285d P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 870fbdcfdac84bacae0ad22dba25f626, State: Initialized, Role: FOLLOWER
I20240501 14:03:55.079893 31394 consensus_queue.cc:260] T a0465b32deb54a5cb18fdd132d33285d P 870fbdcfdac84bacae0ad22dba25f626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:03:55.080379 31394 ts_tablet_manager.cc:1398] T a0465b32deb54a5cb18fdd132d33285d P 870fbdcfdac84bacae0ad22dba25f626: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:03:55.166083 31373 raft_consensus.cc:482] T 04e140965a314fbca43595aed44942b4 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:03:55.166237 31373 raft_consensus.cc:504] T 04e140965a314fbca43595aed44942b4 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:03:55.166860 31373 leader_election.cc:288] T 04e140965a314fbca43595aed44942b4 P 8883498098f4454bbfe563fb83d31899 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9964c1c76dd14f7dbce766b0c022ab51 (127.29.124.134:38101), 870fbdcfdac84bacae0ad22dba25f626 (127.29.124.132:44267)
I20240501 14:03:55.167315 31032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e140965a314fbca43595aed44942b4" candidate_uuid: "8883498098f4454bbfe563fb83d31899" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9964c1c76dd14f7dbce766b0c022ab51" is_pre_election: true
I20240501 14:03:55.167346 30771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e140965a314fbca43595aed44942b4" candidate_uuid: "8883498098f4454bbfe563fb83d31899" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "870fbdcfdac84bacae0ad22dba25f626" is_pre_election: true
I20240501 14:03:55.167501 31032 raft_consensus.cc:2454] T 04e140965a314fbca43595aed44942b4 P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8883498098f4454bbfe563fb83d31899 in term 0.
I20240501 14:03:55.167546 30771 raft_consensus.cc:2454] T 04e140965a314fbca43595aed44942b4 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8883498098f4454bbfe563fb83d31899 in term 0.
I20240501 14:03:55.167924 30442 leader_election.cc:302] T 04e140965a314fbca43595aed44942b4 P 8883498098f4454bbfe563fb83d31899 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8883498098f4454bbfe563fb83d31899, 9964c1c76dd14f7dbce766b0c022ab51; no voters:
I20240501 14:03:55.168200 31373 raft_consensus.cc:2781] T 04e140965a314fbca43595aed44942b4 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:03:55.168301 31373 raft_consensus.cc:482] T 04e140965a314fbca43595aed44942b4 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:03:55.168363 31373 raft_consensus.cc:3037] T 04e140965a314fbca43595aed44942b4 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:55.170158 31373 raft_consensus.cc:504] T 04e140965a314fbca43595aed44942b4 P 8883498098f4454bbfe563fb83d31899 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:03:55.170627 31373 leader_election.cc:288] T 04e140965a314fbca43595aed44942b4 P 8883498098f4454bbfe563fb83d31899 [CANDIDATE]: Term 1 election: Requested vote from peers 9964c1c76dd14f7dbce766b0c022ab51 (127.29.124.134:38101), 870fbdcfdac84bacae0ad22dba25f626 (127.29.124.132:44267)
I20240501 14:03:55.170996 31032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e140965a314fbca43595aed44942b4" candidate_uuid: "8883498098f4454bbfe563fb83d31899" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9964c1c76dd14f7dbce766b0c022ab51"
I20240501 14:03:55.171149 31032 raft_consensus.cc:3037] T 04e140965a314fbca43595aed44942b4 P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:55.171092 30771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04e140965a314fbca43595aed44942b4" candidate_uuid: "8883498098f4454bbfe563fb83d31899" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "870fbdcfdac84bacae0ad22dba25f626"
I20240501 14:03:55.171285 30771 raft_consensus.cc:3037] T 04e140965a314fbca43595aed44942b4 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:55.172879 31032 raft_consensus.cc:2454] T 04e140965a314fbca43595aed44942b4 P 9964c1c76dd14f7dbce766b0c022ab51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8883498098f4454bbfe563fb83d31899 in term 1.
I20240501 14:03:55.173060 30771 raft_consensus.cc:2454] T 04e140965a314fbca43595aed44942b4 P 870fbdcfdac84bacae0ad22dba25f626 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8883498098f4454bbfe563fb83d31899 in term 1.
I20240501 14:03:55.173283 30442 leader_election.cc:302] T 04e140965a314fbca43595aed44942b4 P 8883498098f4454bbfe563fb83d31899 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8883498098f4454bbfe563fb83d31899, 9964c1c76dd14f7dbce766b0c022ab51; no voters:
I20240501 14:03:55.173563 31373 raft_consensus.cc:2781] T 04e140965a314fbca43595aed44942b4 P 8883498098f4454bbfe563fb83d31899 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:03:55.173692 31373 raft_consensus.cc:686] T 04e140965a314fbca43595aed44942b4 P 8883498098f4454bbfe563fb83d31899 [term 1 LEADER]: Becoming Leader. State: Replica: 8883498098f4454bbfe563fb83d31899, State: Running, Role: LEADER
I20240501 14:03:55.174036 31373 consensus_queue.cc:237] T 04e140965a314fbca43595aed44942b4 P 8883498098f4454bbfe563fb83d31899 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:03:55.176754 30237 catalog_manager.cc:5484] T 04e140965a314fbca43595aed44942b4 P 8883498098f4454bbfe563fb83d31899 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8883498098f4454bbfe563fb83d31899 (127.29.124.130). New cstate: current_term: 1 leader_uuid: "8883498098f4454bbfe563fb83d31899" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } health_report { overall_health: UNKNOWN } } }
I20240501 14:03:55.212502 31373 raft_consensus.cc:482] T a0465b32deb54a5cb18fdd132d33285d P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:03:55.212680 31373 raft_consensus.cc:504] T a0465b32deb54a5cb18fdd132d33285d P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:03:55.213280 31373 leader_election.cc:288] T a0465b32deb54a5cb18fdd132d33285d P 8883498098f4454bbfe563fb83d31899 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9964c1c76dd14f7dbce766b0c022ab51 (127.29.124.134:38101), 870fbdcfdac84bacae0ad22dba25f626 (127.29.124.132:44267)
I20240501 14:03:55.213748 31032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0465b32deb54a5cb18fdd132d33285d" candidate_uuid: "8883498098f4454bbfe563fb83d31899" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9964c1c76dd14f7dbce766b0c022ab51" is_pre_election: true
I20240501 14:03:55.213805 30771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0465b32deb54a5cb18fdd132d33285d" candidate_uuid: "8883498098f4454bbfe563fb83d31899" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "870fbdcfdac84bacae0ad22dba25f626" is_pre_election: true
I20240501 14:03:55.213965 31032 raft_consensus.cc:2454] T a0465b32deb54a5cb18fdd132d33285d P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8883498098f4454bbfe563fb83d31899 in term 0.
I20240501 14:03:55.214370 30442 leader_election.cc:302] T a0465b32deb54a5cb18fdd132d33285d P 8883498098f4454bbfe563fb83d31899 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8883498098f4454bbfe563fb83d31899, 9964c1c76dd14f7dbce766b0c022ab51; no voters:
I20240501 14:03:55.214370 30771 raft_consensus.cc:2454] T a0465b32deb54a5cb18fdd132d33285d P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8883498098f4454bbfe563fb83d31899 in term 0.
I20240501 14:03:55.214622 31373 raft_consensus.cc:2781] T a0465b32deb54a5cb18fdd132d33285d P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:03:55.214763 31373 raft_consensus.cc:482] T a0465b32deb54a5cb18fdd132d33285d P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:03:55.214915 31373 raft_consensus.cc:3037] T a0465b32deb54a5cb18fdd132d33285d P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:55.216823 31373 raft_consensus.cc:504] T a0465b32deb54a5cb18fdd132d33285d P 8883498098f4454bbfe563fb83d31899 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:03:55.217670 31032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0465b32deb54a5cb18fdd132d33285d" candidate_uuid: "8883498098f4454bbfe563fb83d31899" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9964c1c76dd14f7dbce766b0c022ab51"
I20240501 14:03:55.217844 31032 raft_consensus.cc:3037] T a0465b32deb54a5cb18fdd132d33285d P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:55.218072 30771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0465b32deb54a5cb18fdd132d33285d" candidate_uuid: "8883498098f4454bbfe563fb83d31899" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "870fbdcfdac84bacae0ad22dba25f626"
I20240501 14:03:55.218245 30771 raft_consensus.cc:3037] T a0465b32deb54a5cb18fdd132d33285d P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:55.219583 31032 raft_consensus.cc:2454] T a0465b32deb54a5cb18fdd132d33285d P 9964c1c76dd14f7dbce766b0c022ab51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8883498098f4454bbfe563fb83d31899 in term 1.
I20240501 14:03:55.219797 31373 leader_election.cc:288] T a0465b32deb54a5cb18fdd132d33285d P 8883498098f4454bbfe563fb83d31899 [CANDIDATE]: Term 1 election: Requested vote from peers 9964c1c76dd14f7dbce766b0c022ab51 (127.29.124.134:38101), 870fbdcfdac84bacae0ad22dba25f626 (127.29.124.132:44267)
I20240501 14:03:55.219991 30442 leader_election.cc:302] T a0465b32deb54a5cb18fdd132d33285d P 8883498098f4454bbfe563fb83d31899 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8883498098f4454bbfe563fb83d31899, 9964c1c76dd14f7dbce766b0c022ab51; no voters:
I20240501 14:03:55.220196 31373 raft_consensus.cc:2781] T a0465b32deb54a5cb18fdd132d33285d P 8883498098f4454bbfe563fb83d31899 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:03:55.220325 31373 raft_consensus.cc:686] T a0465b32deb54a5cb18fdd132d33285d P 8883498098f4454bbfe563fb83d31899 [term 1 LEADER]: Becoming Leader. State: Replica: 8883498098f4454bbfe563fb83d31899, State: Running, Role: LEADER
I20240501 14:03:55.220579 31373 consensus_queue.cc:237] T a0465b32deb54a5cb18fdd132d33285d P 8883498098f4454bbfe563fb83d31899 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:03:55.220824 30771 raft_consensus.cc:2454] T a0465b32deb54a5cb18fdd132d33285d P 870fbdcfdac84bacae0ad22dba25f626 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8883498098f4454bbfe563fb83d31899 in term 1.
I20240501 14:03:55.223234 30237 catalog_manager.cc:5484] T a0465b32deb54a5cb18fdd132d33285d P 8883498098f4454bbfe563fb83d31899 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8883498098f4454bbfe563fb83d31899 (127.29.124.130). New cstate: current_term: 1 leader_uuid: "8883498098f4454bbfe563fb83d31899" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } health_report { overall_health: HEALTHY } } }
I20240501 14:03:55.425510 31304 raft_consensus.cc:482] T dcd50830d6fd4aa3bdcc89c4e60c13ca P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:03:55.425688 31304 raft_consensus.cc:504] T dcd50830d6fd4aa3bdcc89c4e60c13ca P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } }
I20240501 14:03:55.426234 31304 leader_election.cc:288] T dcd50830d6fd4aa3bdcc89c4e60c13ca P 99f6ca2d65d94b33b7a1d2341c24179a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3e62ccad691413ea2554182f73418a3 (127.29.124.131:43965), c0fad54e965647aaaa34b56674646cb4 (127.29.124.129:42363)
I20240501 14:03:55.426739 30638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd50830d6fd4aa3bdcc89c4e60c13ca" candidate_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3e62ccad691413ea2554182f73418a3" is_pre_election: true
I20240501 14:03:55.426743 30378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd50830d6fd4aa3bdcc89c4e60c13ca" candidate_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0fad54e965647aaaa34b56674646cb4" is_pre_election: true
I20240501 14:03:55.426959 30638 raft_consensus.cc:2454] T dcd50830d6fd4aa3bdcc89c4e60c13ca P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99f6ca2d65d94b33b7a1d2341c24179a in term 0.
I20240501 14:03:55.426959 30378 raft_consensus.cc:2454] T dcd50830d6fd4aa3bdcc89c4e60c13ca P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99f6ca2d65d94b33b7a1d2341c24179a in term 0.
I20240501 14:03:55.427376 30835 leader_election.cc:302] T dcd50830d6fd4aa3bdcc89c4e60c13ca P 99f6ca2d65d94b33b7a1d2341c24179a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99f6ca2d65d94b33b7a1d2341c24179a, a3e62ccad691413ea2554182f73418a3; no voters:
I20240501 14:03:55.427670 31304 raft_consensus.cc:2781] T dcd50830d6fd4aa3bdcc89c4e60c13ca P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:03:55.427765 31304 raft_consensus.cc:482] T dcd50830d6fd4aa3bdcc89c4e60c13ca P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:03:55.427863 31304 raft_consensus.cc:3037] T dcd50830d6fd4aa3bdcc89c4e60c13ca P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:55.429692 31304 raft_consensus.cc:504] T dcd50830d6fd4aa3bdcc89c4e60c13ca P 99f6ca2d65d94b33b7a1d2341c24179a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } }
I20240501 14:03:55.430248 31304 leader_election.cc:288] T dcd50830d6fd4aa3bdcc89c4e60c13ca P 99f6ca2d65d94b33b7a1d2341c24179a [CANDIDATE]: Term 1 election: Requested vote from peers a3e62ccad691413ea2554182f73418a3 (127.29.124.131:43965), c0fad54e965647aaaa34b56674646cb4 (127.29.124.129:42363)
I20240501 14:03:55.430572 30638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd50830d6fd4aa3bdcc89c4e60c13ca" candidate_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3e62ccad691413ea2554182f73418a3"
I20240501 14:03:55.430649 30378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dcd50830d6fd4aa3bdcc89c4e60c13ca" candidate_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0fad54e965647aaaa34b56674646cb4"
I20240501 14:03:55.430774 30638 raft_consensus.cc:3037] T dcd50830d6fd4aa3bdcc89c4e60c13ca P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:55.430833 30378 raft_consensus.cc:3037] T dcd50830d6fd4aa3bdcc89c4e60c13ca P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:55.432514 30378 raft_consensus.cc:2454] T dcd50830d6fd4aa3bdcc89c4e60c13ca P c0fad54e965647aaaa34b56674646cb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99f6ca2d65d94b33b7a1d2341c24179a in term 1.
I20240501 14:03:55.432585 30638 raft_consensus.cc:2454] T dcd50830d6fd4aa3bdcc89c4e60c13ca P a3e62ccad691413ea2554182f73418a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99f6ca2d65d94b33b7a1d2341c24179a in term 1.
I20240501 14:03:55.432897 30836 leader_election.cc:302] T dcd50830d6fd4aa3bdcc89c4e60c13ca P 99f6ca2d65d94b33b7a1d2341c24179a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99f6ca2d65d94b33b7a1d2341c24179a, c0fad54e965647aaaa34b56674646cb4; no voters:
I20240501 14:03:55.433156 31304 raft_consensus.cc:2781] T dcd50830d6fd4aa3bdcc89c4e60c13ca P 99f6ca2d65d94b33b7a1d2341c24179a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:03:55.433517 31304 raft_consensus.cc:686] T dcd50830d6fd4aa3bdcc89c4e60c13ca P 99f6ca2d65d94b33b7a1d2341c24179a [term 1 LEADER]: Becoming Leader. State: Replica: 99f6ca2d65d94b33b7a1d2341c24179a, State: Running, Role: LEADER
I20240501 14:03:55.433758 31304 consensus_queue.cc:237] T dcd50830d6fd4aa3bdcc89c4e60c13ca P 99f6ca2d65d94b33b7a1d2341c24179a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } }
I20240501 14:03:55.436367 30237 catalog_manager.cc:5484] T dcd50830d6fd4aa3bdcc89c4e60c13ca P 99f6ca2d65d94b33b7a1d2341c24179a reported cstate change: term changed from 0 to 1, leader changed from <none> to 99f6ca2d65d94b33b7a1d2341c24179a (127.29.124.133). New cstate: current_term: 1 leader_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } health_report { overall_health: UNKNOWN } } }
I20240501 14:03:55.642266 31373 consensus_queue.cc:1035] T a0465b32deb54a5cb18fdd132d33285d P 8883498098f4454bbfe563fb83d31899 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:55.649936 31232 consensus_queue.cc:1035] T a0465b32deb54a5cb18fdd132d33285d P 8883498098f4454bbfe563fb83d31899 [LEADER]: Connected to new peer: Peer: permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:55.748293 31373 consensus_queue.cc:1035] T 04e140965a314fbca43595aed44942b4 P 8883498098f4454bbfe563fb83d31899 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:55.754271 31419 consensus_queue.cc:1035] T 04e140965a314fbca43595aed44942b4 P 8883498098f4454bbfe563fb83d31899 [LEADER]: Connected to new peer: Peer: permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:03:55.797533 31411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:03:55.797813 31411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:03:55.804829 31411 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240501 14:03:55.941026 31304 consensus_queue.cc:1035] T dcd50830d6fd4aa3bdcc89c4e60c13ca P 99f6ca2d65d94b33b7a1d2341c24179a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:55.998483 30378 raft_consensus.cc:1261] T dcd50830d6fd4aa3bdcc89c4e60c13ca P c0fad54e965647aaaa34b56674646cb4 [term 1 FOLLOWER]: Refusing update from remote peer 99f6ca2d65d94b33b7a1d2341c24179a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:03:56.000280 31459 consensus_queue.cc:1035] T dcd50830d6fd4aa3bdcc89c4e60c13ca P 99f6ca2d65d94b33b7a1d2341c24179a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:56.091069 31464 mvcc.cc:205] Tried to move back new op lower bound from 7022887878532186112 to 7022887876338651136. Current Snapshot: MvccSnapshot[applied={T|T < 7022887878532186112}]
INSERT report
rows total: 2000
time total: 243.615 ms
time per row: 0.121807 ms
-----------------------------------------------------
Suppressions used:
count bytes template
4 1192 libcrypto.so.1.1
-----------------------------------------------------
I20240501 14:03:57.044399 30230 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35960:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:03:57.053030 30881 tablet_service.cc:1467] Processing CreateTablet for tablet 47d3e9076bdf4ba394ba574849e78ded (DEFAULT_TABLE table=rebalance_test_table_4 [id=0504cf7adb634704b948e801d9fd7cda]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:57.053313 30619 tablet_service.cc:1467] Processing CreateTablet for tablet 47d3e9076bdf4ba394ba574849e78ded (DEFAULT_TABLE table=rebalance_test_table_4 [id=0504cf7adb634704b948e801d9fd7cda]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:57.053514 30881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47d3e9076bdf4ba394ba574849e78ded. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:57.053686 30619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47d3e9076bdf4ba394ba574849e78ded. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:57.053642 30358 tablet_service.cc:1467] Processing CreateTablet for tablet 47d3e9076bdf4ba394ba574849e78ded (DEFAULT_TABLE table=rebalance_test_table_4 [id=0504cf7adb634704b948e801d9fd7cda]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:57.054082 30358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47d3e9076bdf4ba394ba574849e78ded. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:57.058218 31012 tablet_service.cc:1467] Processing CreateTablet for tablet 7d59b84cb9ed49a58606d439fbd51f07 (DEFAULT_TABLE table=rebalance_test_table_4 [id=0504cf7adb634704b948e801d9fd7cda]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:57.058395 30620 tablet_service.cc:1467] Processing CreateTablet for tablet fc3d4e63dc414a1cbb65d9e0b831c036 (DEFAULT_TABLE table=rebalance_test_table_4 [id=0504cf7adb634704b948e801d9fd7cda]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:57.058404 30882 tablet_service.cc:1467] Processing CreateTablet for tablet fc3d4e63dc414a1cbb65d9e0b831c036 (DEFAULT_TABLE table=rebalance_test_table_4 [id=0504cf7adb634704b948e801d9fd7cda]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:57.058645 31012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d59b84cb9ed49a58606d439fbd51f07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:57.058713 30620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc3d4e63dc414a1cbb65d9e0b831c036. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:57.058729 30882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc3d4e63dc414a1cbb65d9e0b831c036. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:57.059995 30751 tablet_service.cc:1467] Processing CreateTablet for tablet 7d59b84cb9ed49a58606d439fbd51f07 (DEFAULT_TABLE table=rebalance_test_table_4 [id=0504cf7adb634704b948e801d9fd7cda]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:57.060384 30751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d59b84cb9ed49a58606d439fbd51f07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:57.062017 30488 tablet_service.cc:1467] Processing CreateTablet for tablet 7d59b84cb9ed49a58606d439fbd51f07 (DEFAULT_TABLE table=rebalance_test_table_4 [id=0504cf7adb634704b948e801d9fd7cda]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:57.062426 30488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d59b84cb9ed49a58606d439fbd51f07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:57.064903 30357 tablet_service.cc:1467] Processing CreateTablet for tablet fc3d4e63dc414a1cbb65d9e0b831c036 (DEFAULT_TABLE table=rebalance_test_table_4 [id=0504cf7adb634704b948e801d9fd7cda]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:57.065297 30357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc3d4e63dc414a1cbb65d9e0b831c036. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:57.072786 31481 tablet_bootstrap.cc:492] T 47d3e9076bdf4ba394ba574849e78ded P a3e62ccad691413ea2554182f73418a3: Bootstrap starting.
I20240501 14:03:57.076153 31481 tablet_bootstrap.cc:654] T 47d3e9076bdf4ba394ba574849e78ded P a3e62ccad691413ea2554182f73418a3: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:57.080847 31484 tablet_bootstrap.cc:492] T 7d59b84cb9ed49a58606d439fbd51f07 P 870fbdcfdac84bacae0ad22dba25f626: Bootstrap starting.
I20240501 14:03:57.083236 31484 tablet_bootstrap.cc:654] T 7d59b84cb9ed49a58606d439fbd51f07 P 870fbdcfdac84bacae0ad22dba25f626: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:57.087627 31481 tablet_bootstrap.cc:492] T 47d3e9076bdf4ba394ba574849e78ded P a3e62ccad691413ea2554182f73418a3: No bootstrap required, opened a new log
I20240501 14:03:57.088325 31481 ts_tablet_manager.cc:1367] T 47d3e9076bdf4ba394ba574849e78ded P a3e62ccad691413ea2554182f73418a3: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.003s
I20240501 14:03:57.089753 31481 raft_consensus.cc:348] T 47d3e9076bdf4ba394ba574849e78ded P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } }
I20240501 14:03:57.090152 31484 tablet_bootstrap.cc:492] T 7d59b84cb9ed49a58606d439fbd51f07 P 870fbdcfdac84bacae0ad22dba25f626: No bootstrap required, opened a new log
I20240501 14:03:57.090337 31484 ts_tablet_manager.cc:1367] T 7d59b84cb9ed49a58606d439fbd51f07 P 870fbdcfdac84bacae0ad22dba25f626: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20240501 14:03:57.091132 31481 raft_consensus.cc:374] T 47d3e9076bdf4ba394ba574849e78ded P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:57.091137 31484 raft_consensus.cc:348] T 7d59b84cb9ed49a58606d439fbd51f07 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } }
I20240501 14:03:57.091356 31484 raft_consensus.cc:374] T 7d59b84cb9ed49a58606d439fbd51f07 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:57.091461 31484 raft_consensus.cc:729] T 7d59b84cb9ed49a58606d439fbd51f07 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 870fbdcfdac84bacae0ad22dba25f626, State: Initialized, Role: FOLLOWER
I20240501 14:03:57.091607 31481 raft_consensus.cc:729] T 47d3e9076bdf4ba394ba574849e78ded P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3e62ccad691413ea2554182f73418a3, State: Initialized, Role: FOLLOWER
I20240501 14:03:57.091722 31484 consensus_queue.cc:260] T 7d59b84cb9ed49a58606d439fbd51f07 P 870fbdcfdac84bacae0ad22dba25f626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } }
I20240501 14:03:57.092131 31481 consensus_queue.cc:260] T 47d3e9076bdf4ba394ba574849e78ded P a3e62ccad691413ea2554182f73418a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } }
I20240501 14:03:57.092468 31484 ts_tablet_manager.cc:1398] T 7d59b84cb9ed49a58606d439fbd51f07 P 870fbdcfdac84bacae0ad22dba25f626: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:03:57.093710 31481 ts_tablet_manager.cc:1398] T 47d3e9076bdf4ba394ba574849e78ded P a3e62ccad691413ea2554182f73418a3: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:03:57.094329 31481 tablet_bootstrap.cc:492] T fc3d4e63dc414a1cbb65d9e0b831c036 P a3e62ccad691413ea2554182f73418a3: Bootstrap starting.
I20240501 14:03:57.096380 31481 tablet_bootstrap.cc:654] T fc3d4e63dc414a1cbb65d9e0b831c036 P a3e62ccad691413ea2554182f73418a3: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:57.096812 31487 tablet_bootstrap.cc:492] T 7d59b84cb9ed49a58606d439fbd51f07 P 8883498098f4454bbfe563fb83d31899: Bootstrap starting.
I20240501 14:03:57.099015 31487 tablet_bootstrap.cc:654] T 7d59b84cb9ed49a58606d439fbd51f07 P 8883498098f4454bbfe563fb83d31899: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:57.099913 31199 raft_consensus.cc:482] T 7d59b84cb9ed49a58606d439fbd51f07 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:03:57.100090 31199 raft_consensus.cc:504] T 7d59b84cb9ed49a58606d439fbd51f07 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } }
I20240501 14:03:57.100250 31482 tablet_bootstrap.cc:492] T fc3d4e63dc414a1cbb65d9e0b831c036 P 99f6ca2d65d94b33b7a1d2341c24179a: Bootstrap starting.
I20240501 14:03:57.100683 31199 leader_election.cc:288] T 7d59b84cb9ed49a58606d439fbd51f07 P 870fbdcfdac84bacae0ad22dba25f626 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8883498098f4454bbfe563fb83d31899 (127.29.124.130:43099), 9964c1c76dd14f7dbce766b0c022ab51 (127.29.124.134:38101)
I20240501 14:03:57.101022 30509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d59b84cb9ed49a58606d439fbd51f07" candidate_uuid: "870fbdcfdac84bacae0ad22dba25f626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8883498098f4454bbfe563fb83d31899" is_pre_election: true
I20240501 14:03:57.101438 31033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d59b84cb9ed49a58606d439fbd51f07" candidate_uuid: "870fbdcfdac84bacae0ad22dba25f626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9964c1c76dd14f7dbce766b0c022ab51" is_pre_election: true
I20240501 14:03:57.101720 31486 tablet_bootstrap.cc:492] T 7d59b84cb9ed49a58606d439fbd51f07 P 9964c1c76dd14f7dbce766b0c022ab51: Bootstrap starting.
W20240501 14:03:57.101887 30707 leader_election.cc:341] T 7d59b84cb9ed49a58606d439fbd51f07 P 870fbdcfdac84bacae0ad22dba25f626 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8883498098f4454bbfe563fb83d31899 (127.29.124.130:43099): Illegal state: must be running to vote when last-logged opid is not known
W20240501 14:03:57.102219 30704 leader_election.cc:341] T 7d59b84cb9ed49a58606d439fbd51f07 P 870fbdcfdac84bacae0ad22dba25f626 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9964c1c76dd14f7dbce766b0c022ab51 (127.29.124.134:38101): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:03:57.102365 30704 leader_election.cc:302] T 7d59b84cb9ed49a58606d439fbd51f07 P 870fbdcfdac84bacae0ad22dba25f626 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 870fbdcfdac84bacae0ad22dba25f626; no voters: 8883498098f4454bbfe563fb83d31899, 9964c1c76dd14f7dbce766b0c022ab51
I20240501 14:03:57.102558 31481 tablet_bootstrap.cc:492] T fc3d4e63dc414a1cbb65d9e0b831c036 P a3e62ccad691413ea2554182f73418a3: No bootstrap required, opened a new log
I20240501 14:03:57.102643 31199 raft_consensus.cc:2726] T 7d59b84cb9ed49a58606d439fbd51f07 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:03:57.102759 31481 ts_tablet_manager.cc:1367] T fc3d4e63dc414a1cbb65d9e0b831c036 P a3e62ccad691413ea2554182f73418a3: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.001s
I20240501 14:03:57.103292 31486 tablet_bootstrap.cc:654] T 7d59b84cb9ed49a58606d439fbd51f07 P 9964c1c76dd14f7dbce766b0c022ab51: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:57.103417 31481 raft_consensus.cc:348] T fc3d4e63dc414a1cbb65d9e0b831c036 P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:03:57.103657 31481 raft_consensus.cc:374] T fc3d4e63dc414a1cbb65d9e0b831c036 P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:57.103731 31481 raft_consensus.cc:729] T fc3d4e63dc414a1cbb65d9e0b831c036 P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3e62ccad691413ea2554182f73418a3, State: Initialized, Role: FOLLOWER
I20240501 14:03:57.103952 31481 consensus_queue.cc:260] T fc3d4e63dc414a1cbb65d9e0b831c036 P a3e62ccad691413ea2554182f73418a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:03:57.104532 31481 ts_tablet_manager.cc:1398] T fc3d4e63dc414a1cbb65d9e0b831c036 P a3e62ccad691413ea2554182f73418a3: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:03:57.105580 31482 tablet_bootstrap.cc:654] T fc3d4e63dc414a1cbb65d9e0b831c036 P 99f6ca2d65d94b33b7a1d2341c24179a: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:57.107457 31487 tablet_bootstrap.cc:492] T 7d59b84cb9ed49a58606d439fbd51f07 P 8883498098f4454bbfe563fb83d31899: No bootstrap required, opened a new log
I20240501 14:03:57.107641 31487 ts_tablet_manager.cc:1367] T 7d59b84cb9ed49a58606d439fbd51f07 P 8883498098f4454bbfe563fb83d31899: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240501 14:03:57.107774 31483 tablet_bootstrap.cc:492] T fc3d4e63dc414a1cbb65d9e0b831c036 P c0fad54e965647aaaa34b56674646cb4: Bootstrap starting.
I20240501 14:03:57.108546 31487 raft_consensus.cc:348] T 7d59b84cb9ed49a58606d439fbd51f07 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } }
I20240501 14:03:57.108700 31487 raft_consensus.cc:374] T 7d59b84cb9ed49a58606d439fbd51f07 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:57.108814 31487 raft_consensus.cc:729] T 7d59b84cb9ed49a58606d439fbd51f07 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8883498098f4454bbfe563fb83d31899, State: Initialized, Role: FOLLOWER
I20240501 14:03:57.108863 31486 tablet_bootstrap.cc:492] T 7d59b84cb9ed49a58606d439fbd51f07 P 9964c1c76dd14f7dbce766b0c022ab51: No bootstrap required, opened a new log
I20240501 14:03:57.109009 31487 consensus_queue.cc:260] T 7d59b84cb9ed49a58606d439fbd51f07 P 8883498098f4454bbfe563fb83d31899 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } }
I20240501 14:03:57.109090 31486 ts_tablet_manager.cc:1367] T 7d59b84cb9ed49a58606d439fbd51f07 P 9964c1c76dd14f7dbce766b0c022ab51: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240501 14:03:57.109593 31487 ts_tablet_manager.cc:1398] T 7d59b84cb9ed49a58606d439fbd51f07 P 8883498098f4454bbfe563fb83d31899: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:03:57.109776 31486 raft_consensus.cc:348] T 7d59b84cb9ed49a58606d439fbd51f07 P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } }
I20240501 14:03:57.109941 31483 tablet_bootstrap.cc:654] T fc3d4e63dc414a1cbb65d9e0b831c036 P c0fad54e965647aaaa34b56674646cb4: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:57.109984 31486 raft_consensus.cc:374] T 7d59b84cb9ed49a58606d439fbd51f07 P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:57.110076 31486 raft_consensus.cc:729] T 7d59b84cb9ed49a58606d439fbd51f07 P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9964c1c76dd14f7dbce766b0c022ab51, State: Initialized, Role: FOLLOWER
I20240501 14:03:57.110307 31486 consensus_queue.cc:260] T 7d59b84cb9ed49a58606d439fbd51f07 P 9964c1c76dd14f7dbce766b0c022ab51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } }
I20240501 14:03:57.111058 31486 ts_tablet_manager.cc:1398] T 7d59b84cb9ed49a58606d439fbd51f07 P 9964c1c76dd14f7dbce766b0c022ab51: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:03:57.112149 31483 tablet_bootstrap.cc:492] T fc3d4e63dc414a1cbb65d9e0b831c036 P c0fad54e965647aaaa34b56674646cb4: No bootstrap required, opened a new log
I20240501 14:03:57.112289 31483 ts_tablet_manager.cc:1367] T fc3d4e63dc414a1cbb65d9e0b831c036 P c0fad54e965647aaaa34b56674646cb4: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240501 14:03:57.113047 31482 tablet_bootstrap.cc:492] T fc3d4e63dc414a1cbb65d9e0b831c036 P 99f6ca2d65d94b33b7a1d2341c24179a: No bootstrap required, opened a new log
I20240501 14:03:57.113046 31483 raft_consensus.cc:348] T fc3d4e63dc414a1cbb65d9e0b831c036 P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:03:57.113193 31482 ts_tablet_manager.cc:1367] T fc3d4e63dc414a1cbb65d9e0b831c036 P 99f6ca2d65d94b33b7a1d2341c24179a: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20240501 14:03:57.113291 31483 raft_consensus.cc:374] T fc3d4e63dc414a1cbb65d9e0b831c036 P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:57.113391 31483 raft_consensus.cc:729] T fc3d4e63dc414a1cbb65d9e0b831c036 P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0fad54e965647aaaa34b56674646cb4, State: Initialized, Role: FOLLOWER
I20240501 14:03:57.113610 31483 consensus_queue.cc:260] T fc3d4e63dc414a1cbb65d9e0b831c036 P c0fad54e965647aaaa34b56674646cb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:03:57.114311 31483 ts_tablet_manager.cc:1398] T fc3d4e63dc414a1cbb65d9e0b831c036 P c0fad54e965647aaaa34b56674646cb4: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:03:57.114578 31483 tablet_bootstrap.cc:492] T 47d3e9076bdf4ba394ba574849e78ded P c0fad54e965647aaaa34b56674646cb4: Bootstrap starting.
I20240501 14:03:57.113943 31482 raft_consensus.cc:348] T fc3d4e63dc414a1cbb65d9e0b831c036 P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:03:57.114742 31482 raft_consensus.cc:374] T fc3d4e63dc414a1cbb65d9e0b831c036 P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:57.114867 31482 raft_consensus.cc:729] T fc3d4e63dc414a1cbb65d9e0b831c036 P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99f6ca2d65d94b33b7a1d2341c24179a, State: Initialized, Role: FOLLOWER
I20240501 14:03:57.115126 31482 consensus_queue.cc:260] T fc3d4e63dc414a1cbb65d9e0b831c036 P 99f6ca2d65d94b33b7a1d2341c24179a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:03:57.115659 31482 ts_tablet_manager.cc:1398] T fc3d4e63dc414a1cbb65d9e0b831c036 P 99f6ca2d65d94b33b7a1d2341c24179a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:03:57.116407 31482 tablet_bootstrap.cc:492] T 47d3e9076bdf4ba394ba574849e78ded P 99f6ca2d65d94b33b7a1d2341c24179a: Bootstrap starting.
I20240501 14:03:57.117450 31483 tablet_bootstrap.cc:654] T 47d3e9076bdf4ba394ba574849e78ded P c0fad54e965647aaaa34b56674646cb4: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:57.117944 31482 tablet_bootstrap.cc:654] T 47d3e9076bdf4ba394ba574849e78ded P 99f6ca2d65d94b33b7a1d2341c24179a: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:57.119410 31483 tablet_bootstrap.cc:492] T 47d3e9076bdf4ba394ba574849e78ded P c0fad54e965647aaaa34b56674646cb4: No bootstrap required, opened a new log
I20240501 14:03:57.119575 31483 ts_tablet_manager.cc:1367] T 47d3e9076bdf4ba394ba574849e78ded P c0fad54e965647aaaa34b56674646cb4: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240501 14:03:57.119750 31482 tablet_bootstrap.cc:492] T 47d3e9076bdf4ba394ba574849e78ded P 99f6ca2d65d94b33b7a1d2341c24179a: No bootstrap required, opened a new log
I20240501 14:03:57.119892 31482 ts_tablet_manager.cc:1367] T 47d3e9076bdf4ba394ba574849e78ded P 99f6ca2d65d94b33b7a1d2341c24179a: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:03:57.120180 31483 raft_consensus.cc:348] T 47d3e9076bdf4ba394ba574849e78ded P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } }
I20240501 14:03:57.120347 31483 raft_consensus.cc:374] T 47d3e9076bdf4ba394ba574849e78ded P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:57.120421 31483 raft_consensus.cc:729] T 47d3e9076bdf4ba394ba574849e78ded P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0fad54e965647aaaa34b56674646cb4, State: Initialized, Role: FOLLOWER
I20240501 14:03:57.120501 31482 raft_consensus.cc:348] T 47d3e9076bdf4ba394ba574849e78ded P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } }
I20240501 14:03:57.120690 31482 raft_consensus.cc:374] T 47d3e9076bdf4ba394ba574849e78ded P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:57.120659 31483 consensus_queue.cc:260] T 47d3e9076bdf4ba394ba574849e78ded P c0fad54e965647aaaa34b56674646cb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } }
I20240501 14:03:57.120769 31482 raft_consensus.cc:729] T 47d3e9076bdf4ba394ba574849e78ded P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99f6ca2d65d94b33b7a1d2341c24179a, State: Initialized, Role: FOLLOWER
I20240501 14:03:57.121145 31483 ts_tablet_manager.cc:1398] T 47d3e9076bdf4ba394ba574849e78ded P c0fad54e965647aaaa34b56674646cb4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:03:57.121135 31482 consensus_queue.cc:260] T 47d3e9076bdf4ba394ba574849e78ded P 99f6ca2d65d94b33b7a1d2341c24179a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } }
I20240501 14:03:57.121716 31482 ts_tablet_manager.cc:1398] T 47d3e9076bdf4ba394ba574849e78ded P 99f6ca2d65d94b33b7a1d2341c24179a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:03:57.166852 31311 raft_consensus.cc:482] T 47d3e9076bdf4ba394ba574849e78ded P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:03:57.167053 31311 raft_consensus.cc:504] T 47d3e9076bdf4ba394ba574849e78ded P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } }
I20240501 14:03:57.167640 31311 leader_election.cc:288] T 47d3e9076bdf4ba394ba574849e78ded P c0fad54e965647aaaa34b56674646cb4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99f6ca2d65d94b33b7a1d2341c24179a (127.29.124.133:35195), a3e62ccad691413ea2554182f73418a3 (127.29.124.131:43965)
I20240501 14:03:57.168017 30901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47d3e9076bdf4ba394ba574849e78ded" candidate_uuid: "c0fad54e965647aaaa34b56674646cb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" is_pre_election: true
I20240501 14:03:57.168274 30901 raft_consensus.cc:2454] T 47d3e9076bdf4ba394ba574849e78ded P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0fad54e965647aaaa34b56674646cb4 in term 0.
I20240501 14:03:57.168217 30638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47d3e9076bdf4ba394ba574849e78ded" candidate_uuid: "c0fad54e965647aaaa34b56674646cb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3e62ccad691413ea2554182f73418a3" is_pre_election: true
I20240501 14:03:57.168450 30638 raft_consensus.cc:2454] T 47d3e9076bdf4ba394ba574849e78ded P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0fad54e965647aaaa34b56674646cb4 in term 0.
I20240501 14:03:57.168726 30313 leader_election.cc:302] T 47d3e9076bdf4ba394ba574849e78ded P c0fad54e965647aaaa34b56674646cb4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99f6ca2d65d94b33b7a1d2341c24179a, c0fad54e965647aaaa34b56674646cb4; no voters:
I20240501 14:03:57.169044 31311 raft_consensus.cc:2781] T 47d3e9076bdf4ba394ba574849e78ded P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:03:57.169157 31311 raft_consensus.cc:482] T 47d3e9076bdf4ba394ba574849e78ded P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:03:57.169283 31311 raft_consensus.cc:3037] T 47d3e9076bdf4ba394ba574849e78ded P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:57.171252 31311 raft_consensus.cc:504] T 47d3e9076bdf4ba394ba574849e78ded P c0fad54e965647aaaa34b56674646cb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } }
I20240501 14:03:57.171797 31311 leader_election.cc:288] T 47d3e9076bdf4ba394ba574849e78ded P c0fad54e965647aaaa34b56674646cb4 [CANDIDATE]: Term 1 election: Requested vote from peers 99f6ca2d65d94b33b7a1d2341c24179a (127.29.124.133:35195), a3e62ccad691413ea2554182f73418a3 (127.29.124.131:43965)
I20240501 14:03:57.172191 30638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47d3e9076bdf4ba394ba574849e78ded" candidate_uuid: "c0fad54e965647aaaa34b56674646cb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3e62ccad691413ea2554182f73418a3"
I20240501 14:03:57.172187 30901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47d3e9076bdf4ba394ba574849e78ded" candidate_uuid: "c0fad54e965647aaaa34b56674646cb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99f6ca2d65d94b33b7a1d2341c24179a"
I20240501 14:03:57.172369 30638 raft_consensus.cc:3037] T 47d3e9076bdf4ba394ba574849e78ded P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:57.172391 30901 raft_consensus.cc:3037] T 47d3e9076bdf4ba394ba574849e78ded P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:57.175065 30901 raft_consensus.cc:2454] T 47d3e9076bdf4ba394ba574849e78ded P 99f6ca2d65d94b33b7a1d2341c24179a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0fad54e965647aaaa34b56674646cb4 in term 1.
I20240501 14:03:57.175089 30638 raft_consensus.cc:2454] T 47d3e9076bdf4ba394ba574849e78ded P a3e62ccad691413ea2554182f73418a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0fad54e965647aaaa34b56674646cb4 in term 1.
I20240501 14:03:57.175441 30313 leader_election.cc:302] T 47d3e9076bdf4ba394ba574849e78ded P c0fad54e965647aaaa34b56674646cb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99f6ca2d65d94b33b7a1d2341c24179a, c0fad54e965647aaaa34b56674646cb4; no voters:
I20240501 14:03:57.175714 31311 raft_consensus.cc:2781] T 47d3e9076bdf4ba394ba574849e78ded P c0fad54e965647aaaa34b56674646cb4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:03:57.176162 31311 raft_consensus.cc:686] T 47d3e9076bdf4ba394ba574849e78ded P c0fad54e965647aaaa34b56674646cb4 [term 1 LEADER]: Becoming Leader. State: Replica: c0fad54e965647aaaa34b56674646cb4, State: Running, Role: LEADER
I20240501 14:03:57.176664 31311 consensus_queue.cc:237] T 47d3e9076bdf4ba394ba574849e78ded P c0fad54e965647aaaa34b56674646cb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } }
I20240501 14:03:57.179630 30234 catalog_manager.cc:5484] T 47d3e9076bdf4ba394ba574849e78ded P c0fad54e965647aaaa34b56674646cb4 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0fad54e965647aaaa34b56674646cb4 (127.29.124.129). New cstate: current_term: 1 leader_uuid: "c0fad54e965647aaaa34b56674646cb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } health_report { overall_health: HEALTHY } } }
I20240501 14:03:57.228787 31456 raft_consensus.cc:482] T fc3d4e63dc414a1cbb65d9e0b831c036 P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:03:57.228998 31456 raft_consensus.cc:504] T fc3d4e63dc414a1cbb65d9e0b831c036 P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:03:57.229785 31456 leader_election.cc:288] T fc3d4e63dc414a1cbb65d9e0b831c036 P 99f6ca2d65d94b33b7a1d2341c24179a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0fad54e965647aaaa34b56674646cb4 (127.29.124.129:42363), a3e62ccad691413ea2554182f73418a3 (127.29.124.131:43965)
I20240501 14:03:57.230300 30638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc3d4e63dc414a1cbb65d9e0b831c036" candidate_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3e62ccad691413ea2554182f73418a3" is_pre_election: true
I20240501 14:03:57.230311 30378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc3d4e63dc414a1cbb65d9e0b831c036" candidate_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0fad54e965647aaaa34b56674646cb4" is_pre_election: true
I20240501 14:03:57.230551 30378 raft_consensus.cc:2454] T fc3d4e63dc414a1cbb65d9e0b831c036 P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99f6ca2d65d94b33b7a1d2341c24179a in term 0.
I20240501 14:03:57.230549 30638 raft_consensus.cc:2454] T fc3d4e63dc414a1cbb65d9e0b831c036 P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99f6ca2d65d94b33b7a1d2341c24179a in term 0.
I20240501 14:03:57.230999 30836 leader_election.cc:302] T fc3d4e63dc414a1cbb65d9e0b831c036 P 99f6ca2d65d94b33b7a1d2341c24179a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99f6ca2d65d94b33b7a1d2341c24179a, c0fad54e965647aaaa34b56674646cb4; no voters:
I20240501 14:03:57.231313 31456 raft_consensus.cc:2781] T fc3d4e63dc414a1cbb65d9e0b831c036 P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:03:57.231433 31456 raft_consensus.cc:482] T fc3d4e63dc414a1cbb65d9e0b831c036 P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:03:57.231520 31456 raft_consensus.cc:3037] T fc3d4e63dc414a1cbb65d9e0b831c036 P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:57.233829 31456 raft_consensus.cc:504] T fc3d4e63dc414a1cbb65d9e0b831c036 P 99f6ca2d65d94b33b7a1d2341c24179a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:03:57.234395 31456 leader_election.cc:288] T fc3d4e63dc414a1cbb65d9e0b831c036 P 99f6ca2d65d94b33b7a1d2341c24179a [CANDIDATE]: Term 1 election: Requested vote from peers c0fad54e965647aaaa34b56674646cb4 (127.29.124.129:42363), a3e62ccad691413ea2554182f73418a3 (127.29.124.131:43965)
I20240501 14:03:57.234808 30378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc3d4e63dc414a1cbb65d9e0b831c036" candidate_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0fad54e965647aaaa34b56674646cb4"
I20240501 14:03:57.234823 30638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc3d4e63dc414a1cbb65d9e0b831c036" candidate_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3e62ccad691413ea2554182f73418a3"
I20240501 14:03:57.235028 30378 raft_consensus.cc:3037] T fc3d4e63dc414a1cbb65d9e0b831c036 P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:57.235059 30638 raft_consensus.cc:3037] T fc3d4e63dc414a1cbb65d9e0b831c036 P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:57.237692 30378 raft_consensus.cc:2454] T fc3d4e63dc414a1cbb65d9e0b831c036 P c0fad54e965647aaaa34b56674646cb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99f6ca2d65d94b33b7a1d2341c24179a in term 1.
I20240501 14:03:57.237692 30638 raft_consensus.cc:2454] T fc3d4e63dc414a1cbb65d9e0b831c036 P a3e62ccad691413ea2554182f73418a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99f6ca2d65d94b33b7a1d2341c24179a in term 1.
I20240501 14:03:57.238111 30836 leader_election.cc:302] T fc3d4e63dc414a1cbb65d9e0b831c036 P 99f6ca2d65d94b33b7a1d2341c24179a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99f6ca2d65d94b33b7a1d2341c24179a, c0fad54e965647aaaa34b56674646cb4; no voters:
I20240501 14:03:57.238410 31456 raft_consensus.cc:2781] T fc3d4e63dc414a1cbb65d9e0b831c036 P 99f6ca2d65d94b33b7a1d2341c24179a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:03:57.238795 31456 raft_consensus.cc:686] T fc3d4e63dc414a1cbb65d9e0b831c036 P 99f6ca2d65d94b33b7a1d2341c24179a [term 1 LEADER]: Becoming Leader. State: Replica: 99f6ca2d65d94b33b7a1d2341c24179a, State: Running, Role: LEADER
I20240501 14:03:57.239136 31456 consensus_queue.cc:237] T fc3d4e63dc414a1cbb65d9e0b831c036 P 99f6ca2d65d94b33b7a1d2341c24179a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:03:57.242111 30234 catalog_manager.cc:5484] T fc3d4e63dc414a1cbb65d9e0b831c036 P 99f6ca2d65d94b33b7a1d2341c24179a reported cstate change: term changed from 0 to 1, leader changed from <none> to 99f6ca2d65d94b33b7a1d2341c24179a (127.29.124.133). New cstate: current_term: 1 leader_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } health_report { overall_health: UNKNOWN } } }
I20240501 14:03:57.344856 31232 raft_consensus.cc:482] T 7d59b84cb9ed49a58606d439fbd51f07 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:03:57.345078 31232 raft_consensus.cc:504] T 7d59b84cb9ed49a58606d439fbd51f07 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } }
I20240501 14:03:57.345742 31232 leader_election.cc:288] T 7d59b84cb9ed49a58606d439fbd51f07 P 8883498098f4454bbfe563fb83d31899 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 870fbdcfdac84bacae0ad22dba25f626 (127.29.124.132:44267), 9964c1c76dd14f7dbce766b0c022ab51 (127.29.124.134:38101)
I20240501 14:03:57.346314 31033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d59b84cb9ed49a58606d439fbd51f07" candidate_uuid: "8883498098f4454bbfe563fb83d31899" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9964c1c76dd14f7dbce766b0c022ab51" is_pre_election: true
I20240501 14:03:57.346326 30770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d59b84cb9ed49a58606d439fbd51f07" candidate_uuid: "8883498098f4454bbfe563fb83d31899" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "870fbdcfdac84bacae0ad22dba25f626" is_pre_election: true
I20240501 14:03:57.346552 30770 raft_consensus.cc:2454] T 7d59b84cb9ed49a58606d439fbd51f07 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8883498098f4454bbfe563fb83d31899 in term 0.
I20240501 14:03:57.346575 31033 raft_consensus.cc:2454] T 7d59b84cb9ed49a58606d439fbd51f07 P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8883498098f4454bbfe563fb83d31899 in term 0.
I20240501 14:03:57.346967 30442 leader_election.cc:302] T 7d59b84cb9ed49a58606d439fbd51f07 P 8883498098f4454bbfe563fb83d31899 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8883498098f4454bbfe563fb83d31899, 9964c1c76dd14f7dbce766b0c022ab51; no voters:
I20240501 14:03:57.347276 31232 raft_consensus.cc:2781] T 7d59b84cb9ed49a58606d439fbd51f07 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:03:57.347404 31232 raft_consensus.cc:482] T 7d59b84cb9ed49a58606d439fbd51f07 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:03:57.347493 31232 raft_consensus.cc:3037] T 7d59b84cb9ed49a58606d439fbd51f07 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:57.349350 31232 raft_consensus.cc:504] T 7d59b84cb9ed49a58606d439fbd51f07 P 8883498098f4454bbfe563fb83d31899 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } }
I20240501 14:03:57.349884 31232 leader_election.cc:288] T 7d59b84cb9ed49a58606d439fbd51f07 P 8883498098f4454bbfe563fb83d31899 [CANDIDATE]: Term 1 election: Requested vote from peers 870fbdcfdac84bacae0ad22dba25f626 (127.29.124.132:44267), 9964c1c76dd14f7dbce766b0c022ab51 (127.29.124.134:38101)
I20240501 14:03:57.350323 31033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d59b84cb9ed49a58606d439fbd51f07" candidate_uuid: "8883498098f4454bbfe563fb83d31899" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9964c1c76dd14f7dbce766b0c022ab51"
I20240501 14:03:57.350323 30770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d59b84cb9ed49a58606d439fbd51f07" candidate_uuid: "8883498098f4454bbfe563fb83d31899" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "870fbdcfdac84bacae0ad22dba25f626"
I20240501 14:03:57.350519 30770 raft_consensus.cc:3037] T 7d59b84cb9ed49a58606d439fbd51f07 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:57.350530 31033 raft_consensus.cc:3037] T 7d59b84cb9ed49a58606d439fbd51f07 P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:57.352464 31033 raft_consensus.cc:2454] T 7d59b84cb9ed49a58606d439fbd51f07 P 9964c1c76dd14f7dbce766b0c022ab51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8883498098f4454bbfe563fb83d31899 in term 1.
I20240501 14:03:57.353245 30442 leader_election.cc:302] T 7d59b84cb9ed49a58606d439fbd51f07 P 8883498098f4454bbfe563fb83d31899 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8883498098f4454bbfe563fb83d31899, 9964c1c76dd14f7dbce766b0c022ab51; no voters:
I20240501 14:03:57.353542 31232 raft_consensus.cc:2781] T 7d59b84cb9ed49a58606d439fbd51f07 P 8883498098f4454bbfe563fb83d31899 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:03:57.353927 31232 raft_consensus.cc:686] T 7d59b84cb9ed49a58606d439fbd51f07 P 8883498098f4454bbfe563fb83d31899 [term 1 LEADER]: Becoming Leader. State: Replica: 8883498098f4454bbfe563fb83d31899, State: Running, Role: LEADER
I20240501 14:03:57.354224 31232 consensus_queue.cc:237] T 7d59b84cb9ed49a58606d439fbd51f07 P 8883498098f4454bbfe563fb83d31899 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } }
I20240501 14:03:57.354368 30770 raft_consensus.cc:2454] T 7d59b84cb9ed49a58606d439fbd51f07 P 870fbdcfdac84bacae0ad22dba25f626 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8883498098f4454bbfe563fb83d31899 in term 1.
I20240501 14:03:57.359221 30234 catalog_manager.cc:5484] T 7d59b84cb9ed49a58606d439fbd51f07 P 8883498098f4454bbfe563fb83d31899 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8883498098f4454bbfe563fb83d31899 (127.29.124.130). New cstate: current_term: 1 leader_uuid: "8883498098f4454bbfe563fb83d31899" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } health_report { overall_health: UNKNOWN } } }
I20240501 14:03:57.619375 31311 consensus_queue.cc:1035] T 47d3e9076bdf4ba394ba574849e78ded P c0fad54e965647aaaa34b56674646cb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:57.627086 31311 consensus_queue.cc:1035] T 47d3e9076bdf4ba394ba574849e78ded P c0fad54e965647aaaa34b56674646cb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:03:57.677887 31503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:03:57.678215 31503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:03:57.685631 31503 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:03:57.708034 31456 consensus_queue.cc:1035] T fc3d4e63dc414a1cbb65d9e0b831c036 P 99f6ca2d65d94b33b7a1d2341c24179a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:57.712838 31456 consensus_queue.cc:1035] T fc3d4e63dc414a1cbb65d9e0b831c036 P 99f6ca2d65d94b33b7a1d2341c24179a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_4'
I20240501 14:03:57.854583 31033 raft_consensus.cc:1261] T 7d59b84cb9ed49a58606d439fbd51f07 P 9964c1c76dd14f7dbce766b0c022ab51 [term 1 FOLLOWER]: Refusing update from remote peer 8883498098f4454bbfe563fb83d31899: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:03:57.855383 31232 consensus_queue.cc:1035] T 7d59b84cb9ed49a58606d439fbd51f07 P 8883498098f4454bbfe563fb83d31899 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:57.869015 30770 raft_consensus.cc:1261] T 7d59b84cb9ed49a58606d439fbd51f07 P 870fbdcfdac84bacae0ad22dba25f626 [term 1 FOLLOWER]: Refusing update from remote peer 8883498098f4454bbfe563fb83d31899: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:03:57.869761 31232 consensus_queue.cc:1035] T 7d59b84cb9ed49a58606d439fbd51f07 P 8883498098f4454bbfe563fb83d31899 [LEADER]: Connected to new peer: Peer: permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:57.927148 31552 mvcc.cc:205] Tried to move back new op lower bound from 7022887886234603520 to 7022887884205121536. Current Snapshot: MvccSnapshot[applied={T|T < 7022887886234603520}]
I20240501 14:03:57.941716 31556 mvcc.cc:205] Tried to move back new op lower bound from 7022887886234603520 to 7022887884205121536. Current Snapshot: MvccSnapshot[applied={T|T < 7022887886234603520}]
INSERT report
rows total: 2000
time total: 250.551 ms
time per row: 0.125276 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240501 14:03:58.960728 30228 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35960:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:03:58.970625 30488 tablet_service.cc:1467] Processing CreateTablet for tablet 68614ecd77ea4d39a47045bc46d0f1ba (DEFAULT_TABLE table=rebalance_test_table_5 [id=b6d20202bb274a1aac157c5f2cc6d0e1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:58.970927 30489 tablet_service.cc:1467] Processing CreateTablet for tablet 3e8c0ae4ee104fd5baf3cf14688656e1 (DEFAULT_TABLE table=rebalance_test_table_5 [id=b6d20202bb274a1aac157c5f2cc6d0e1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:58.971107 30488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68614ecd77ea4d39a47045bc46d0f1ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:58.971308 30489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e8c0ae4ee104fd5baf3cf14688656e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:58.972729 30357 tablet_service.cc:1467] Processing CreateTablet for tablet 98d4243714a6471b9636b9bd0aad25f2 (DEFAULT_TABLE table=rebalance_test_table_5 [id=b6d20202bb274a1aac157c5f2cc6d0e1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:58.973134 30357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98d4243714a6471b9636b9bd0aad25f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:58.973870 31012 tablet_service.cc:1467] Processing CreateTablet for tablet 68614ecd77ea4d39a47045bc46d0f1ba (DEFAULT_TABLE table=rebalance_test_table_5 [id=b6d20202bb274a1aac157c5f2cc6d0e1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:58.974293 31012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68614ecd77ea4d39a47045bc46d0f1ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:58.974982 31013 tablet_service.cc:1467] Processing CreateTablet for tablet 3e8c0ae4ee104fd5baf3cf14688656e1 (DEFAULT_TABLE table=rebalance_test_table_5 [id=b6d20202bb274a1aac157c5f2cc6d0e1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:58.975324 31013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e8c0ae4ee104fd5baf3cf14688656e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:58.977653 30751 tablet_service.cc:1467] Processing CreateTablet for tablet 68614ecd77ea4d39a47045bc46d0f1ba (DEFAULT_TABLE table=rebalance_test_table_5 [id=b6d20202bb274a1aac157c5f2cc6d0e1]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:58.977756 30882 tablet_service.cc:1467] Processing CreateTablet for tablet 98d4243714a6471b9636b9bd0aad25f2 (DEFAULT_TABLE table=rebalance_test_table_5 [id=b6d20202bb274a1aac157c5f2cc6d0e1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:58.978050 30751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68614ecd77ea4d39a47045bc46d0f1ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:58.978148 30882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98d4243714a6471b9636b9bd0aad25f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:58.981611 30750 tablet_service.cc:1467] Processing CreateTablet for tablet 3e8c0ae4ee104fd5baf3cf14688656e1 (DEFAULT_TABLE table=rebalance_test_table_5 [id=b6d20202bb274a1aac157c5f2cc6d0e1]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:58.981922 30750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e8c0ae4ee104fd5baf3cf14688656e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:58.985105 31578 tablet_bootstrap.cc:492] T 3e8c0ae4ee104fd5baf3cf14688656e1 P 8883498098f4454bbfe563fb83d31899: Bootstrap starting.
I20240501 14:03:58.985941 30619 tablet_service.cc:1467] Processing CreateTablet for tablet 98d4243714a6471b9636b9bd0aad25f2 (DEFAULT_TABLE table=rebalance_test_table_5 [id=b6d20202bb274a1aac157c5f2cc6d0e1]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:03:58.986359 30619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 98d4243714a6471b9636b9bd0aad25f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:03:58.986852 31578 tablet_bootstrap.cc:654] T 3e8c0ae4ee104fd5baf3cf14688656e1 P 8883498098f4454bbfe563fb83d31899: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:58.990149 31579 tablet_bootstrap.cc:492] T 98d4243714a6471b9636b9bd0aad25f2 P c0fad54e965647aaaa34b56674646cb4: Bootstrap starting.
I20240501 14:03:58.991511 31579 tablet_bootstrap.cc:654] T 98d4243714a6471b9636b9bd0aad25f2 P c0fad54e965647aaaa34b56674646cb4: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:58.995605 31579 tablet_bootstrap.cc:492] T 98d4243714a6471b9636b9bd0aad25f2 P c0fad54e965647aaaa34b56674646cb4: No bootstrap required, opened a new log
I20240501 14:03:58.995743 31579 ts_tablet_manager.cc:1367] T 98d4243714a6471b9636b9bd0aad25f2 P c0fad54e965647aaaa34b56674646cb4: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:03:58.996300 31579 raft_consensus.cc:348] T 98d4243714a6471b9636b9bd0aad25f2 P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } }
I20240501 14:03:58.996485 31579 raft_consensus.cc:374] T 98d4243714a6471b9636b9bd0aad25f2 P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:58.996575 31579 raft_consensus.cc:729] T 98d4243714a6471b9636b9bd0aad25f2 P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0fad54e965647aaaa34b56674646cb4, State: Initialized, Role: FOLLOWER
I20240501 14:03:58.996778 31579 consensus_queue.cc:260] T 98d4243714a6471b9636b9bd0aad25f2 P c0fad54e965647aaaa34b56674646cb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } }
I20240501 14:03:58.997236 31579 ts_tablet_manager.cc:1398] T 98d4243714a6471b9636b9bd0aad25f2 P c0fad54e965647aaaa34b56674646cb4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:03:59.005465 31584 tablet_bootstrap.cc:492] T 68614ecd77ea4d39a47045bc46d0f1ba P 870fbdcfdac84bacae0ad22dba25f626: Bootstrap starting.
I20240501 14:03:59.008355 31584 tablet_bootstrap.cc:654] T 68614ecd77ea4d39a47045bc46d0f1ba P 870fbdcfdac84bacae0ad22dba25f626: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:59.012421 31585 tablet_bootstrap.cc:492] T 98d4243714a6471b9636b9bd0aad25f2 P a3e62ccad691413ea2554182f73418a3: Bootstrap starting.
I20240501 14:03:59.013121 31580 tablet_bootstrap.cc:492] T 98d4243714a6471b9636b9bd0aad25f2 P 99f6ca2d65d94b33b7a1d2341c24179a: Bootstrap starting.
I20240501 14:03:59.013665 31581 tablet_bootstrap.cc:492] T 68614ecd77ea4d39a47045bc46d0f1ba P 9964c1c76dd14f7dbce766b0c022ab51: Bootstrap starting.
I20240501 14:03:59.014034 31585 tablet_bootstrap.cc:654] T 98d4243714a6471b9636b9bd0aad25f2 P a3e62ccad691413ea2554182f73418a3: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:59.014853 31578 tablet_bootstrap.cc:492] T 3e8c0ae4ee104fd5baf3cf14688656e1 P 8883498098f4454bbfe563fb83d31899: No bootstrap required, opened a new log
I20240501 14:03:59.015054 31578 ts_tablet_manager.cc:1367] T 3e8c0ae4ee104fd5baf3cf14688656e1 P 8883498098f4454bbfe563fb83d31899: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.020s
I20240501 14:03:59.016237 31578 raft_consensus.cc:348] T 3e8c0ae4ee104fd5baf3cf14688656e1 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:03:59.016563 31578 raft_consensus.cc:374] T 3e8c0ae4ee104fd5baf3cf14688656e1 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:59.016698 31578 raft_consensus.cc:729] T 3e8c0ae4ee104fd5baf3cf14688656e1 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8883498098f4454bbfe563fb83d31899, State: Initialized, Role: FOLLOWER
I20240501 14:03:59.017002 31578 consensus_queue.cc:260] T 3e8c0ae4ee104fd5baf3cf14688656e1 P 8883498098f4454bbfe563fb83d31899 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:03:59.017130 31581 tablet_bootstrap.cc:654] T 68614ecd77ea4d39a47045bc46d0f1ba P 9964c1c76dd14f7dbce766b0c022ab51: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:59.017792 31578 ts_tablet_manager.cc:1398] T 3e8c0ae4ee104fd5baf3cf14688656e1 P 8883498098f4454bbfe563fb83d31899: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:03:59.018100 31578 tablet_bootstrap.cc:492] T 68614ecd77ea4d39a47045bc46d0f1ba P 8883498098f4454bbfe563fb83d31899: Bootstrap starting.
I20240501 14:03:59.018307 31580 tablet_bootstrap.cc:654] T 98d4243714a6471b9636b9bd0aad25f2 P 99f6ca2d65d94b33b7a1d2341c24179a: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:59.019739 31578 tablet_bootstrap.cc:654] T 68614ecd77ea4d39a47045bc46d0f1ba P 8883498098f4454bbfe563fb83d31899: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:59.020231 31584 tablet_bootstrap.cc:492] T 68614ecd77ea4d39a47045bc46d0f1ba P 870fbdcfdac84bacae0ad22dba25f626: No bootstrap required, opened a new log
I20240501 14:03:59.020368 31584 ts_tablet_manager.cc:1367] T 68614ecd77ea4d39a47045bc46d0f1ba P 870fbdcfdac84bacae0ad22dba25f626: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.009s
I20240501 14:03:59.020455 31585 tablet_bootstrap.cc:492] T 98d4243714a6471b9636b9bd0aad25f2 P a3e62ccad691413ea2554182f73418a3: No bootstrap required, opened a new log
I20240501 14:03:59.021137 31584 raft_consensus.cc:348] T 68614ecd77ea4d39a47045bc46d0f1ba P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:03:59.021379 31584 raft_consensus.cc:374] T 68614ecd77ea4d39a47045bc46d0f1ba P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:59.021468 31584 raft_consensus.cc:729] T 68614ecd77ea4d39a47045bc46d0f1ba P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 870fbdcfdac84bacae0ad22dba25f626, State: Initialized, Role: FOLLOWER
I20240501 14:03:59.021720 31584 consensus_queue.cc:260] T 68614ecd77ea4d39a47045bc46d0f1ba P 870fbdcfdac84bacae0ad22dba25f626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:03:59.022281 31584 ts_tablet_manager.cc:1398] T 68614ecd77ea4d39a47045bc46d0f1ba P 870fbdcfdac84bacae0ad22dba25f626: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:03:59.020620 31585 ts_tablet_manager.cc:1367] T 98d4243714a6471b9636b9bd0aad25f2 P a3e62ccad691413ea2554182f73418a3: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20240501 14:03:59.022650 31584 tablet_bootstrap.cc:492] T 3e8c0ae4ee104fd5baf3cf14688656e1 P 870fbdcfdac84bacae0ad22dba25f626: Bootstrap starting.
I20240501 14:03:59.023263 31585 raft_consensus.cc:348] T 98d4243714a6471b9636b9bd0aad25f2 P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } }
I20240501 14:03:59.023533 31585 raft_consensus.cc:374] T 98d4243714a6471b9636b9bd0aad25f2 P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:59.023655 31585 raft_consensus.cc:729] T 98d4243714a6471b9636b9bd0aad25f2 P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3e62ccad691413ea2554182f73418a3, State: Initialized, Role: FOLLOWER
I20240501 14:03:59.023933 31585 consensus_queue.cc:260] T 98d4243714a6471b9636b9bd0aad25f2 P a3e62ccad691413ea2554182f73418a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } }
I20240501 14:03:59.024125 31584 tablet_bootstrap.cc:654] T 3e8c0ae4ee104fd5baf3cf14688656e1 P 870fbdcfdac84bacae0ad22dba25f626: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:59.025563 31585 ts_tablet_manager.cc:1398] T 98d4243714a6471b9636b9bd0aad25f2 P a3e62ccad691413ea2554182f73418a3: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:03:59.030337 31578 tablet_bootstrap.cc:492] T 68614ecd77ea4d39a47045bc46d0f1ba P 8883498098f4454bbfe563fb83d31899: No bootstrap required, opened a new log
I20240501 14:03:59.030521 31578 ts_tablet_manager.cc:1367] T 68614ecd77ea4d39a47045bc46d0f1ba P 8883498098f4454bbfe563fb83d31899: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.003s
I20240501 14:03:59.031360 31578 raft_consensus.cc:348] T 68614ecd77ea4d39a47045bc46d0f1ba P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:03:59.031595 31578 raft_consensus.cc:374] T 68614ecd77ea4d39a47045bc46d0f1ba P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:59.031673 31578 raft_consensus.cc:729] T 68614ecd77ea4d39a47045bc46d0f1ba P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8883498098f4454bbfe563fb83d31899, State: Initialized, Role: FOLLOWER
I20240501 14:03:59.031893 31578 consensus_queue.cc:260] T 68614ecd77ea4d39a47045bc46d0f1ba P 8883498098f4454bbfe563fb83d31899 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:03:59.032490 31578 ts_tablet_manager.cc:1398] T 68614ecd77ea4d39a47045bc46d0f1ba P 8883498098f4454bbfe563fb83d31899: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240501 14:03:59.032528 31584 tablet_bootstrap.cc:492] T 3e8c0ae4ee104fd5baf3cf14688656e1 P 870fbdcfdac84bacae0ad22dba25f626: No bootstrap required, opened a new log
I20240501 14:03:59.032665 31584 ts_tablet_manager.cc:1367] T 3e8c0ae4ee104fd5baf3cf14688656e1 P 870fbdcfdac84bacae0ad22dba25f626: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20240501 14:03:59.033118 31581 tablet_bootstrap.cc:492] T 68614ecd77ea4d39a47045bc46d0f1ba P 9964c1c76dd14f7dbce766b0c022ab51: No bootstrap required, opened a new log
I20240501 14:03:59.033231 31581 ts_tablet_manager.cc:1367] T 68614ecd77ea4d39a47045bc46d0f1ba P 9964c1c76dd14f7dbce766b0c022ab51: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.003s
I20240501 14:03:59.033398 31584 raft_consensus.cc:348] T 3e8c0ae4ee104fd5baf3cf14688656e1 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:03:59.033612 31584 raft_consensus.cc:374] T 3e8c0ae4ee104fd5baf3cf14688656e1 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:59.033679 31584 raft_consensus.cc:729] T 3e8c0ae4ee104fd5baf3cf14688656e1 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 870fbdcfdac84bacae0ad22dba25f626, State: Initialized, Role: FOLLOWER
I20240501 14:03:59.033936 31584 consensus_queue.cc:260] T 3e8c0ae4ee104fd5baf3cf14688656e1 P 870fbdcfdac84bacae0ad22dba25f626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:03:59.034387 31584 ts_tablet_manager.cc:1398] T 3e8c0ae4ee104fd5baf3cf14688656e1 P 870fbdcfdac84bacae0ad22dba25f626: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:03:59.033977 31581 raft_consensus.cc:348] T 68614ecd77ea4d39a47045bc46d0f1ba P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:03:59.035241 31581 raft_consensus.cc:374] T 68614ecd77ea4d39a47045bc46d0f1ba P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:59.035319 31581 raft_consensus.cc:729] T 68614ecd77ea4d39a47045bc46d0f1ba P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9964c1c76dd14f7dbce766b0c022ab51, State: Initialized, Role: FOLLOWER
I20240501 14:03:59.035494 31581 consensus_queue.cc:260] T 68614ecd77ea4d39a47045bc46d0f1ba P 9964c1c76dd14f7dbce766b0c022ab51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:03:59.035694 31580 tablet_bootstrap.cc:492] T 98d4243714a6471b9636b9bd0aad25f2 P 99f6ca2d65d94b33b7a1d2341c24179a: No bootstrap required, opened a new log
I20240501 14:03:59.035840 31580 ts_tablet_manager.cc:1367] T 98d4243714a6471b9636b9bd0aad25f2 P 99f6ca2d65d94b33b7a1d2341c24179a: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.000s
I20240501 14:03:59.036180 31581 ts_tablet_manager.cc:1398] T 68614ecd77ea4d39a47045bc46d0f1ba P 9964c1c76dd14f7dbce766b0c022ab51: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:03:59.036512 31581 tablet_bootstrap.cc:492] T 3e8c0ae4ee104fd5baf3cf14688656e1 P 9964c1c76dd14f7dbce766b0c022ab51: Bootstrap starting.
I20240501 14:03:59.036561 31580 raft_consensus.cc:348] T 98d4243714a6471b9636b9bd0aad25f2 P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } }
I20240501 14:03:59.036760 31580 raft_consensus.cc:374] T 98d4243714a6471b9636b9bd0aad25f2 P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:59.036849 31580 raft_consensus.cc:729] T 98d4243714a6471b9636b9bd0aad25f2 P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99f6ca2d65d94b33b7a1d2341c24179a, State: Initialized, Role: FOLLOWER
I20240501 14:03:59.037070 31580 consensus_queue.cc:260] T 98d4243714a6471b9636b9bd0aad25f2 P 99f6ca2d65d94b33b7a1d2341c24179a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } }
I20240501 14:03:59.037827 31580 ts_tablet_manager.cc:1398] T 98d4243714a6471b9636b9bd0aad25f2 P 99f6ca2d65d94b33b7a1d2341c24179a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:03:59.038681 31581 tablet_bootstrap.cc:654] T 3e8c0ae4ee104fd5baf3cf14688656e1 P 9964c1c76dd14f7dbce766b0c022ab51: Neither blocks nor log segments found. Creating new log.
I20240501 14:03:59.040663 31581 tablet_bootstrap.cc:492] T 3e8c0ae4ee104fd5baf3cf14688656e1 P 9964c1c76dd14f7dbce766b0c022ab51: No bootstrap required, opened a new log
I20240501 14:03:59.040822 31581 ts_tablet_manager.cc:1367] T 3e8c0ae4ee104fd5baf3cf14688656e1 P 9964c1c76dd14f7dbce766b0c022ab51: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240501 14:03:59.041550 31581 raft_consensus.cc:348] T 3e8c0ae4ee104fd5baf3cf14688656e1 P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:03:59.041708 31581 raft_consensus.cc:374] T 3e8c0ae4ee104fd5baf3cf14688656e1 P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:03:59.041762 31581 raft_consensus.cc:729] T 3e8c0ae4ee104fd5baf3cf14688656e1 P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9964c1c76dd14f7dbce766b0c022ab51, State: Initialized, Role: FOLLOWER
I20240501 14:03:59.041921 31581 consensus_queue.cc:260] T 3e8c0ae4ee104fd5baf3cf14688656e1 P 9964c1c76dd14f7dbce766b0c022ab51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:03:59.042308 31581 ts_tablet_manager.cc:1398] T 3e8c0ae4ee104fd5baf3cf14688656e1 P 9964c1c76dd14f7dbce766b0c022ab51: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20240501 14:03:59.094421 31554 raft_consensus.cc:482] T 68614ecd77ea4d39a47045bc46d0f1ba P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:03:59.094573 31554 raft_consensus.cc:504] T 68614ecd77ea4d39a47045bc46d0f1ba P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:03:59.095224 31554 leader_election.cc:288] T 68614ecd77ea4d39a47045bc46d0f1ba P 8883498098f4454bbfe563fb83d31899 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9964c1c76dd14f7dbce766b0c022ab51 (127.29.124.134:38101), 870fbdcfdac84bacae0ad22dba25f626 (127.29.124.132:44267)
I20240501 14:03:59.095621 31033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68614ecd77ea4d39a47045bc46d0f1ba" candidate_uuid: "8883498098f4454bbfe563fb83d31899" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9964c1c76dd14f7dbce766b0c022ab51" is_pre_election: true
I20240501 14:03:59.095741 30771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68614ecd77ea4d39a47045bc46d0f1ba" candidate_uuid: "8883498098f4454bbfe563fb83d31899" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "870fbdcfdac84bacae0ad22dba25f626" is_pre_election: true
I20240501 14:03:59.095852 31033 raft_consensus.cc:2454] T 68614ecd77ea4d39a47045bc46d0f1ba P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8883498098f4454bbfe563fb83d31899 in term 0.
I20240501 14:03:59.095980 30771 raft_consensus.cc:2454] T 68614ecd77ea4d39a47045bc46d0f1ba P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8883498098f4454bbfe563fb83d31899 in term 0.
I20240501 14:03:59.096284 30442 leader_election.cc:302] T 68614ecd77ea4d39a47045bc46d0f1ba P 8883498098f4454bbfe563fb83d31899 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8883498098f4454bbfe563fb83d31899, 9964c1c76dd14f7dbce766b0c022ab51; no voters:
I20240501 14:03:59.096576 31554 raft_consensus.cc:2781] T 68614ecd77ea4d39a47045bc46d0f1ba P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:03:59.096696 31554 raft_consensus.cc:482] T 68614ecd77ea4d39a47045bc46d0f1ba P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:03:59.096786 31554 raft_consensus.cc:3037] T 68614ecd77ea4d39a47045bc46d0f1ba P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:59.098740 31554 raft_consensus.cc:504] T 68614ecd77ea4d39a47045bc46d0f1ba P 8883498098f4454bbfe563fb83d31899 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:03:59.099309 31554 leader_election.cc:288] T 68614ecd77ea4d39a47045bc46d0f1ba P 8883498098f4454bbfe563fb83d31899 [CANDIDATE]: Term 1 election: Requested vote from peers 9964c1c76dd14f7dbce766b0c022ab51 (127.29.124.134:38101), 870fbdcfdac84bacae0ad22dba25f626 (127.29.124.132:44267)
I20240501 14:03:59.099669 31033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68614ecd77ea4d39a47045bc46d0f1ba" candidate_uuid: "8883498098f4454bbfe563fb83d31899" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9964c1c76dd14f7dbce766b0c022ab51"
I20240501 14:03:59.099754 30771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68614ecd77ea4d39a47045bc46d0f1ba" candidate_uuid: "8883498098f4454bbfe563fb83d31899" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "870fbdcfdac84bacae0ad22dba25f626"
I20240501 14:03:59.099879 31033 raft_consensus.cc:3037] T 68614ecd77ea4d39a47045bc46d0f1ba P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:59.099936 30771 raft_consensus.cc:3037] T 68614ecd77ea4d39a47045bc46d0f1ba P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:59.102550 30771 raft_consensus.cc:2454] T 68614ecd77ea4d39a47045bc46d0f1ba P 870fbdcfdac84bacae0ad22dba25f626 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8883498098f4454bbfe563fb83d31899 in term 1.
I20240501 14:03:59.102550 31033 raft_consensus.cc:2454] T 68614ecd77ea4d39a47045bc46d0f1ba P 9964c1c76dd14f7dbce766b0c022ab51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8883498098f4454bbfe563fb83d31899 in term 1.
I20240501 14:03:59.102913 30445 leader_election.cc:302] T 68614ecd77ea4d39a47045bc46d0f1ba P 8883498098f4454bbfe563fb83d31899 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 870fbdcfdac84bacae0ad22dba25f626, 8883498098f4454bbfe563fb83d31899; no voters:
I20240501 14:03:59.103284 31554 raft_consensus.cc:2781] T 68614ecd77ea4d39a47045bc46d0f1ba P 8883498098f4454bbfe563fb83d31899 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:03:59.103615 31554 raft_consensus.cc:686] T 68614ecd77ea4d39a47045bc46d0f1ba P 8883498098f4454bbfe563fb83d31899 [term 1 LEADER]: Becoming Leader. State: Replica: 8883498098f4454bbfe563fb83d31899, State: Running, Role: LEADER
I20240501 14:03:59.103861 31554 consensus_queue.cc:237] T 68614ecd77ea4d39a47045bc46d0f1ba P 8883498098f4454bbfe563fb83d31899 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:03:59.106606 30237 catalog_manager.cc:5484] T 68614ecd77ea4d39a47045bc46d0f1ba P 8883498098f4454bbfe563fb83d31899 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8883498098f4454bbfe563fb83d31899 (127.29.124.130). New cstate: current_term: 1 leader_uuid: "8883498098f4454bbfe563fb83d31899" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } health_report { overall_health: UNKNOWN } } }
I20240501 14:03:59.312182 31497 raft_consensus.cc:482] T 98d4243714a6471b9636b9bd0aad25f2 P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:03:59.312371 31497 raft_consensus.cc:504] T 98d4243714a6471b9636b9bd0aad25f2 P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } }
I20240501 14:03:59.312991 31497 leader_election.cc:288] T 98d4243714a6471b9636b9bd0aad25f2 P c0fad54e965647aaaa34b56674646cb4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99f6ca2d65d94b33b7a1d2341c24179a (127.29.124.133:35195), a3e62ccad691413ea2554182f73418a3 (127.29.124.131:43965)
I20240501 14:03:59.313431 30901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98d4243714a6471b9636b9bd0aad25f2" candidate_uuid: "c0fad54e965647aaaa34b56674646cb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" is_pre_election: true
I20240501 14:03:59.313527 30639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98d4243714a6471b9636b9bd0aad25f2" candidate_uuid: "c0fad54e965647aaaa34b56674646cb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3e62ccad691413ea2554182f73418a3" is_pre_election: true
I20240501 14:03:59.313634 30901 raft_consensus.cc:2454] T 98d4243714a6471b9636b9bd0aad25f2 P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0fad54e965647aaaa34b56674646cb4 in term 0.
I20240501 14:03:59.313758 30639 raft_consensus.cc:2454] T 98d4243714a6471b9636b9bd0aad25f2 P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0fad54e965647aaaa34b56674646cb4 in term 0.
I20240501 14:03:59.314054 30313 leader_election.cc:302] T 98d4243714a6471b9636b9bd0aad25f2 P c0fad54e965647aaaa34b56674646cb4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99f6ca2d65d94b33b7a1d2341c24179a, c0fad54e965647aaaa34b56674646cb4; no voters:
I20240501 14:03:59.314339 31497 raft_consensus.cc:2781] T 98d4243714a6471b9636b9bd0aad25f2 P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:03:59.314445 31497 raft_consensus.cc:482] T 98d4243714a6471b9636b9bd0aad25f2 P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:03:59.314525 31497 raft_consensus.cc:3037] T 98d4243714a6471b9636b9bd0aad25f2 P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:59.317546 31497 raft_consensus.cc:504] T 98d4243714a6471b9636b9bd0aad25f2 P c0fad54e965647aaaa34b56674646cb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } }
I20240501 14:03:59.318094 31497 leader_election.cc:288] T 98d4243714a6471b9636b9bd0aad25f2 P c0fad54e965647aaaa34b56674646cb4 [CANDIDATE]: Term 1 election: Requested vote from peers 99f6ca2d65d94b33b7a1d2341c24179a (127.29.124.133:35195), a3e62ccad691413ea2554182f73418a3 (127.29.124.131:43965)
I20240501 14:03:59.318503 30901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98d4243714a6471b9636b9bd0aad25f2" candidate_uuid: "c0fad54e965647aaaa34b56674646cb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99f6ca2d65d94b33b7a1d2341c24179a"
I20240501 14:03:59.318506 30639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "98d4243714a6471b9636b9bd0aad25f2" candidate_uuid: "c0fad54e965647aaaa34b56674646cb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3e62ccad691413ea2554182f73418a3"
I20240501 14:03:59.318665 30901 raft_consensus.cc:3037] T 98d4243714a6471b9636b9bd0aad25f2 P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:59.318660 30639 raft_consensus.cc:3037] T 98d4243714a6471b9636b9bd0aad25f2 P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:59.320497 30901 raft_consensus.cc:2454] T 98d4243714a6471b9636b9bd0aad25f2 P 99f6ca2d65d94b33b7a1d2341c24179a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0fad54e965647aaaa34b56674646cb4 in term 1.
I20240501 14:03:59.320714 30639 raft_consensus.cc:2454] T 98d4243714a6471b9636b9bd0aad25f2 P a3e62ccad691413ea2554182f73418a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0fad54e965647aaaa34b56674646cb4 in term 1.
I20240501 14:03:59.320937 30313 leader_election.cc:302] T 98d4243714a6471b9636b9bd0aad25f2 P c0fad54e965647aaaa34b56674646cb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99f6ca2d65d94b33b7a1d2341c24179a, c0fad54e965647aaaa34b56674646cb4; no voters:
I20240501 14:03:59.321285 31497 raft_consensus.cc:2781] T 98d4243714a6471b9636b9bd0aad25f2 P c0fad54e965647aaaa34b56674646cb4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:03:59.321688 31497 raft_consensus.cc:686] T 98d4243714a6471b9636b9bd0aad25f2 P c0fad54e965647aaaa34b56674646cb4 [term 1 LEADER]: Becoming Leader. State: Replica: c0fad54e965647aaaa34b56674646cb4, State: Running, Role: LEADER
I20240501 14:03:59.322099 31497 consensus_queue.cc:237] T 98d4243714a6471b9636b9bd0aad25f2 P c0fad54e965647aaaa34b56674646cb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } }
I20240501 14:03:59.325009 30237 catalog_manager.cc:5484] T 98d4243714a6471b9636b9bd0aad25f2 P c0fad54e965647aaaa34b56674646cb4 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0fad54e965647aaaa34b56674646cb4 (127.29.124.129). New cstate: current_term: 1 leader_uuid: "c0fad54e965647aaaa34b56674646cb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } health_report { overall_health: HEALTHY } } }
I20240501 14:03:59.449923 31554 raft_consensus.cc:482] T 3e8c0ae4ee104fd5baf3cf14688656e1 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:03:59.450132 31554 raft_consensus.cc:504] T 3e8c0ae4ee104fd5baf3cf14688656e1 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:03:59.450778 31554 leader_election.cc:288] T 3e8c0ae4ee104fd5baf3cf14688656e1 P 8883498098f4454bbfe563fb83d31899 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9964c1c76dd14f7dbce766b0c022ab51 (127.29.124.134:38101), 870fbdcfdac84bacae0ad22dba25f626 (127.29.124.132:44267)
I20240501 14:03:59.451234 31033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e8c0ae4ee104fd5baf3cf14688656e1" candidate_uuid: "8883498098f4454bbfe563fb83d31899" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9964c1c76dd14f7dbce766b0c022ab51" is_pre_election: true
I20240501 14:03:59.451287 30771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e8c0ae4ee104fd5baf3cf14688656e1" candidate_uuid: "8883498098f4454bbfe563fb83d31899" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "870fbdcfdac84bacae0ad22dba25f626" is_pre_election: true
I20240501 14:03:59.451435 31033 raft_consensus.cc:2454] T 3e8c0ae4ee104fd5baf3cf14688656e1 P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8883498098f4454bbfe563fb83d31899 in term 0.
I20240501 14:03:59.451565 30771 raft_consensus.cc:2454] T 3e8c0ae4ee104fd5baf3cf14688656e1 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8883498098f4454bbfe563fb83d31899 in term 0.
I20240501 14:03:59.451817 30442 leader_election.cc:302] T 3e8c0ae4ee104fd5baf3cf14688656e1 P 8883498098f4454bbfe563fb83d31899 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8883498098f4454bbfe563fb83d31899, 9964c1c76dd14f7dbce766b0c022ab51; no voters:
I20240501 14:03:59.452121 31554 raft_consensus.cc:2781] T 3e8c0ae4ee104fd5baf3cf14688656e1 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:03:59.452204 31554 raft_consensus.cc:482] T 3e8c0ae4ee104fd5baf3cf14688656e1 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:03:59.452313 31554 raft_consensus.cc:3037] T 3e8c0ae4ee104fd5baf3cf14688656e1 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:59.454408 31554 raft_consensus.cc:504] T 3e8c0ae4ee104fd5baf3cf14688656e1 P 8883498098f4454bbfe563fb83d31899 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:03:59.454965 31554 leader_election.cc:288] T 3e8c0ae4ee104fd5baf3cf14688656e1 P 8883498098f4454bbfe563fb83d31899 [CANDIDATE]: Term 1 election: Requested vote from peers 9964c1c76dd14f7dbce766b0c022ab51 (127.29.124.134:38101), 870fbdcfdac84bacae0ad22dba25f626 (127.29.124.132:44267)
I20240501 14:03:59.455322 31033 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e8c0ae4ee104fd5baf3cf14688656e1" candidate_uuid: "8883498098f4454bbfe563fb83d31899" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9964c1c76dd14f7dbce766b0c022ab51"
I20240501 14:03:59.455400 30771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e8c0ae4ee104fd5baf3cf14688656e1" candidate_uuid: "8883498098f4454bbfe563fb83d31899" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "870fbdcfdac84bacae0ad22dba25f626"
I20240501 14:03:59.455487 31033 raft_consensus.cc:3037] T 3e8c0ae4ee104fd5baf3cf14688656e1 P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:59.455564 30771 raft_consensus.cc:3037] T 3e8c0ae4ee104fd5baf3cf14688656e1 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:03:59.457299 31033 raft_consensus.cc:2454] T 3e8c0ae4ee104fd5baf3cf14688656e1 P 9964c1c76dd14f7dbce766b0c022ab51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8883498098f4454bbfe563fb83d31899 in term 1.
I20240501 14:03:59.457361 30771 raft_consensus.cc:2454] T 3e8c0ae4ee104fd5baf3cf14688656e1 P 870fbdcfdac84bacae0ad22dba25f626 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8883498098f4454bbfe563fb83d31899 in term 1.
I20240501 14:03:59.457715 30442 leader_election.cc:302] T 3e8c0ae4ee104fd5baf3cf14688656e1 P 8883498098f4454bbfe563fb83d31899 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8883498098f4454bbfe563fb83d31899, 9964c1c76dd14f7dbce766b0c022ab51; no voters:
I20240501 14:03:59.457952 31554 raft_consensus.cc:2781] T 3e8c0ae4ee104fd5baf3cf14688656e1 P 8883498098f4454bbfe563fb83d31899 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:03:59.458119 31554 raft_consensus.cc:686] T 3e8c0ae4ee104fd5baf3cf14688656e1 P 8883498098f4454bbfe563fb83d31899 [term 1 LEADER]: Becoming Leader. State: Replica: 8883498098f4454bbfe563fb83d31899, State: Running, Role: LEADER
I20240501 14:03:59.458354 31554 consensus_queue.cc:237] T 3e8c0ae4ee104fd5baf3cf14688656e1 P 8883498098f4454bbfe563fb83d31899 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:03:59.460853 30237 catalog_manager.cc:5484] T 3e8c0ae4ee104fd5baf3cf14688656e1 P 8883498098f4454bbfe563fb83d31899 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8883498098f4454bbfe563fb83d31899 (127.29.124.130). New cstate: current_term: 1 leader_uuid: "8883498098f4454bbfe563fb83d31899" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } health_report { overall_health: UNKNOWN } } }
I20240501 14:03:59.507879 31554 consensus_queue.cc:1035] T 68614ecd77ea4d39a47045bc46d0f1ba P 8883498098f4454bbfe563fb83d31899 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:59.513146 31554 consensus_queue.cc:1035] T 68614ecd77ea4d39a47045bc46d0f1ba P 8883498098f4454bbfe563fb83d31899 [LEADER]: Connected to new peer: Peer: permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:59.824508 31596 consensus_queue.cc:1035] T 98d4243714a6471b9636b9bd0aad25f2 P c0fad54e965647aaaa34b56674646cb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:03:59.835659 31596 consensus_queue.cc:1035] T 98d4243714a6471b9636b9bd0aad25f2 P c0fad54e965647aaaa34b56674646cb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:03:59.862767 31605 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:03:59.863094 31605 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:03:59.870349 31605 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240501 14:04:00.014058 31033 raft_consensus.cc:1261] T 3e8c0ae4ee104fd5baf3cf14688656e1 P 9964c1c76dd14f7dbce766b0c022ab51 [term 1 FOLLOWER]: Refusing update from remote peer 8883498098f4454bbfe563fb83d31899: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:04:00.014986 31554 consensus_queue.cc:1035] T 3e8c0ae4ee104fd5baf3cf14688656e1 P 8883498098f4454bbfe563fb83d31899 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:04:00.050264 30771 raft_consensus.cc:1261] T 3e8c0ae4ee104fd5baf3cf14688656e1 P 870fbdcfdac84bacae0ad22dba25f626 [term 1 FOLLOWER]: Refusing update from remote peer 8883498098f4454bbfe563fb83d31899: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:04:00.107848 31658 consensus_queue.cc:1035] T 3e8c0ae4ee104fd5baf3cf14688656e1 P 8883498098f4454bbfe563fb83d31899 [LEADER]: Connected to new peer: Peer: permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 251.821 ms
time per row: 0.12591 ms
I20240501 14:04:00.262841 30228 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35960:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:04:00.271536 30882 tablet_service.cc:1467] Processing CreateTablet for tablet 8db48542aec94b11842b21e51dd330c0 (DEFAULT_TABLE table=rebalance_test_table_6 [id=13508d3de9a34098a5bbda817d3929e9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:00.271945 30882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8db48542aec94b11842b21e51dd330c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:00.272537 30357 tablet_service.cc:1467] Processing CreateTablet for tablet 8db48542aec94b11842b21e51dd330c0 (DEFAULT_TABLE table=rebalance_test_table_6 [id=13508d3de9a34098a5bbda817d3929e9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:00.272773 30881 tablet_service.cc:1467] Processing CreateTablet for tablet 9051190ef3684094b383b9535e6aea83 (DEFAULT_TABLE table=rebalance_test_table_6 [id=13508d3de9a34098a5bbda817d3929e9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:00.272934 30357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8db48542aec94b11842b21e51dd330c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:00.273159 30881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9051190ef3684094b383b9535e6aea83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:00.277309 30619 tablet_service.cc:1467] Processing CreateTablet for tablet 8db48542aec94b11842b21e51dd330c0 (DEFAULT_TABLE table=rebalance_test_table_6 [id=13508d3de9a34098a5bbda817d3929e9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:00.277334 31012 tablet_service.cc:1467] Processing CreateTablet for tablet e445777c9d99413a84d7546320cc621d (DEFAULT_TABLE table=rebalance_test_table_6 [id=13508d3de9a34098a5bbda817d3929e9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:00.277688 30619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8db48542aec94b11842b21e51dd330c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:00.277565 30488 tablet_service.cc:1467] Processing CreateTablet for tablet e445777c9d99413a84d7546320cc621d (DEFAULT_TABLE table=rebalance_test_table_6 [id=13508d3de9a34098a5bbda817d3929e9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:00.277735 31012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e445777c9d99413a84d7546320cc621d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:00.277967 30488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e445777c9d99413a84d7546320cc621d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:00.281692 30620 tablet_service.cc:1467] Processing CreateTablet for tablet 9051190ef3684094b383b9535e6aea83 (DEFAULT_TABLE table=rebalance_test_table_6 [id=13508d3de9a34098a5bbda817d3929e9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:00.282039 30620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9051190ef3684094b383b9535e6aea83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:00.282161 30751 tablet_service.cc:1467] Processing CreateTablet for tablet e445777c9d99413a84d7546320cc621d (DEFAULT_TABLE table=rebalance_test_table_6 [id=13508d3de9a34098a5bbda817d3929e9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:00.282573 30751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e445777c9d99413a84d7546320cc621d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:00.286005 30358 tablet_service.cc:1467] Processing CreateTablet for tablet 9051190ef3684094b383b9535e6aea83 (DEFAULT_TABLE table=rebalance_test_table_6 [id=13508d3de9a34098a5bbda817d3929e9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:00.286361 30358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9051190ef3684094b383b9535e6aea83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:00.291309 31675 tablet_bootstrap.cc:492] T 8db48542aec94b11842b21e51dd330c0 P 99f6ca2d65d94b33b7a1d2341c24179a: Bootstrap starting.
I20240501 14:04:00.293077 31675 tablet_bootstrap.cc:654] T 8db48542aec94b11842b21e51dd330c0 P 99f6ca2d65d94b33b7a1d2341c24179a: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:00.294451 31676 tablet_bootstrap.cc:492] T 8db48542aec94b11842b21e51dd330c0 P c0fad54e965647aaaa34b56674646cb4: Bootstrap starting.
I20240501 14:04:00.295156 31678 tablet_bootstrap.cc:492] T 9051190ef3684094b383b9535e6aea83 P a3e62ccad691413ea2554182f73418a3: Bootstrap starting.
I20240501 14:04:00.295157 31680 tablet_bootstrap.cc:492] T e445777c9d99413a84d7546320cc621d P 9964c1c76dd14f7dbce766b0c022ab51: Bootstrap starting.
I20240501 14:04:00.296815 31680 tablet_bootstrap.cc:654] T e445777c9d99413a84d7546320cc621d P 9964c1c76dd14f7dbce766b0c022ab51: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:00.297181 31678 tablet_bootstrap.cc:654] T 9051190ef3684094b383b9535e6aea83 P a3e62ccad691413ea2554182f73418a3: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:00.299388 31676 tablet_bootstrap.cc:654] T 8db48542aec94b11842b21e51dd330c0 P c0fad54e965647aaaa34b56674646cb4: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:00.301463 31679 tablet_bootstrap.cc:492] T e445777c9d99413a84d7546320cc621d P 870fbdcfdac84bacae0ad22dba25f626: Bootstrap starting.
I20240501 14:04:00.304442 31679 tablet_bootstrap.cc:654] T e445777c9d99413a84d7546320cc621d P 870fbdcfdac84bacae0ad22dba25f626: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:00.305778 31677 tablet_bootstrap.cc:492] T e445777c9d99413a84d7546320cc621d P 8883498098f4454bbfe563fb83d31899: Bootstrap starting.
I20240501 14:04:00.306234 31675 tablet_bootstrap.cc:492] T 8db48542aec94b11842b21e51dd330c0 P 99f6ca2d65d94b33b7a1d2341c24179a: No bootstrap required, opened a new log
I20240501 14:04:00.306424 31675 ts_tablet_manager.cc:1367] T 8db48542aec94b11842b21e51dd330c0 P 99f6ca2d65d94b33b7a1d2341c24179a: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.007s
I20240501 14:04:00.307243 31675 raft_consensus.cc:348] T 8db48542aec94b11842b21e51dd330c0 P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:04:00.307503 31677 tablet_bootstrap.cc:654] T e445777c9d99413a84d7546320cc621d P 8883498098f4454bbfe563fb83d31899: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:00.308423 31675 raft_consensus.cc:374] T 8db48542aec94b11842b21e51dd330c0 P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:00.308604 31680 tablet_bootstrap.cc:492] T e445777c9d99413a84d7546320cc621d P 9964c1c76dd14f7dbce766b0c022ab51: No bootstrap required, opened a new log
I20240501 14:04:00.308605 31675 raft_consensus.cc:729] T 8db48542aec94b11842b21e51dd330c0 P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99f6ca2d65d94b33b7a1d2341c24179a, State: Initialized, Role: FOLLOWER
I20240501 14:04:00.308849 31680 ts_tablet_manager.cc:1367] T e445777c9d99413a84d7546320cc621d P 9964c1c76dd14f7dbce766b0c022ab51: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.004s
I20240501 14:04:00.308866 31675 consensus_queue.cc:260] T 8db48542aec94b11842b21e51dd330c0 P 99f6ca2d65d94b33b7a1d2341c24179a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:04:00.309993 31680 raft_consensus.cc:348] T e445777c9d99413a84d7546320cc621d P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:04:00.310272 31680 raft_consensus.cc:374] T e445777c9d99413a84d7546320cc621d P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:00.310403 31680 raft_consensus.cc:729] T e445777c9d99413a84d7546320cc621d P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9964c1c76dd14f7dbce766b0c022ab51, State: Initialized, Role: FOLLOWER
I20240501 14:04:00.310540 31679 tablet_bootstrap.cc:492] T e445777c9d99413a84d7546320cc621d P 870fbdcfdac84bacae0ad22dba25f626: No bootstrap required, opened a new log
I20240501 14:04:00.310742 31679 ts_tablet_manager.cc:1367] T e445777c9d99413a84d7546320cc621d P 870fbdcfdac84bacae0ad22dba25f626: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.003s
I20240501 14:04:00.310736 31680 consensus_queue.cc:260] T e445777c9d99413a84d7546320cc621d P 9964c1c76dd14f7dbce766b0c022ab51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:04:00.311434 31680 ts_tablet_manager.cc:1398] T e445777c9d99413a84d7546320cc621d P 9964c1c76dd14f7dbce766b0c022ab51: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:04:00.311527 31679 raft_consensus.cc:348] T e445777c9d99413a84d7546320cc621d P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:04:00.312119 31679 raft_consensus.cc:374] T e445777c9d99413a84d7546320cc621d P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:00.312263 31679 raft_consensus.cc:729] T e445777c9d99413a84d7546320cc621d P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 870fbdcfdac84bacae0ad22dba25f626, State: Initialized, Role: FOLLOWER
I20240501 14:04:00.312472 31676 tablet_bootstrap.cc:492] T 8db48542aec94b11842b21e51dd330c0 P c0fad54e965647aaaa34b56674646cb4: No bootstrap required, opened a new log
I20240501 14:04:00.312667 31676 ts_tablet_manager.cc:1367] T 8db48542aec94b11842b21e51dd330c0 P c0fad54e965647aaaa34b56674646cb4: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.005s
I20240501 14:04:00.312587 31679 consensus_queue.cc:260] T e445777c9d99413a84d7546320cc621d P 870fbdcfdac84bacae0ad22dba25f626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:04:00.312728 31675 ts_tablet_manager.cc:1398] T 8db48542aec94b11842b21e51dd330c0 P 99f6ca2d65d94b33b7a1d2341c24179a: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240501 14:04:00.313514 31676 raft_consensus.cc:348] T 8db48542aec94b11842b21e51dd330c0 P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:04:00.313815 31676 raft_consensus.cc:374] T 8db48542aec94b11842b21e51dd330c0 P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:00.313850 31675 tablet_bootstrap.cc:492] T 9051190ef3684094b383b9535e6aea83 P 99f6ca2d65d94b33b7a1d2341c24179a: Bootstrap starting.
I20240501 14:04:00.313982 31676 raft_consensus.cc:729] T 8db48542aec94b11842b21e51dd330c0 P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0fad54e965647aaaa34b56674646cb4, State: Initialized, Role: FOLLOWER
I20240501 14:04:00.314278 31676 consensus_queue.cc:260] T 8db48542aec94b11842b21e51dd330c0 P c0fad54e965647aaaa34b56674646cb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:04:00.315367 31676 ts_tablet_manager.cc:1398] T 8db48542aec94b11842b21e51dd330c0 P c0fad54e965647aaaa34b56674646cb4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:04:00.315716 31676 tablet_bootstrap.cc:492] T 9051190ef3684094b383b9535e6aea83 P c0fad54e965647aaaa34b56674646cb4: Bootstrap starting.
I20240501 14:04:00.317248 31676 tablet_bootstrap.cc:654] T 9051190ef3684094b383b9535e6aea83 P c0fad54e965647aaaa34b56674646cb4: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:00.317288 31675 tablet_bootstrap.cc:654] T 9051190ef3684094b383b9535e6aea83 P 99f6ca2d65d94b33b7a1d2341c24179a: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:00.319213 31679 ts_tablet_manager.cc:1398] T e445777c9d99413a84d7546320cc621d P 870fbdcfdac84bacae0ad22dba25f626: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240501 14:04:00.320607 31678 tablet_bootstrap.cc:492] T 9051190ef3684094b383b9535e6aea83 P a3e62ccad691413ea2554182f73418a3: No bootstrap required, opened a new log
I20240501 14:04:00.320755 31678 ts_tablet_manager.cc:1367] T 9051190ef3684094b383b9535e6aea83 P a3e62ccad691413ea2554182f73418a3: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.014s
I20240501 14:04:00.321596 31678 raft_consensus.cc:348] T 9051190ef3684094b383b9535e6aea83 P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } }
I20240501 14:04:00.321813 31678 raft_consensus.cc:374] T 9051190ef3684094b383b9535e6aea83 P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:00.321913 31678 raft_consensus.cc:729] T 9051190ef3684094b383b9535e6aea83 P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3e62ccad691413ea2554182f73418a3, State: Initialized, Role: FOLLOWER
I20240501 14:04:00.322139 31678 consensus_queue.cc:260] T 9051190ef3684094b383b9535e6aea83 P a3e62ccad691413ea2554182f73418a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } }
I20240501 14:04:00.322661 31678 ts_tablet_manager.cc:1398] T 9051190ef3684094b383b9535e6aea83 P a3e62ccad691413ea2554182f73418a3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:04:00.322674 31676 tablet_bootstrap.cc:492] T 9051190ef3684094b383b9535e6aea83 P c0fad54e965647aaaa34b56674646cb4: No bootstrap required, opened a new log
I20240501 14:04:00.322813 31676 ts_tablet_manager.cc:1367] T 9051190ef3684094b383b9535e6aea83 P c0fad54e965647aaaa34b56674646cb4: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.002s
I20240501 14:04:00.322930 31678 tablet_bootstrap.cc:492] T 8db48542aec94b11842b21e51dd330c0 P a3e62ccad691413ea2554182f73418a3: Bootstrap starting.
I20240501 14:04:00.323565 31676 raft_consensus.cc:348] T 9051190ef3684094b383b9535e6aea83 P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } }
I20240501 14:04:00.323807 31676 raft_consensus.cc:374] T 9051190ef3684094b383b9535e6aea83 P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:00.323923 31676 raft_consensus.cc:729] T 9051190ef3684094b383b9535e6aea83 P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0fad54e965647aaaa34b56674646cb4, State: Initialized, Role: FOLLOWER
I20240501 14:04:00.324224 31675 tablet_bootstrap.cc:492] T 9051190ef3684094b383b9535e6aea83 P 99f6ca2d65d94b33b7a1d2341c24179a: No bootstrap required, opened a new log
I20240501 14:04:00.324366 31678 tablet_bootstrap.cc:654] T 8db48542aec94b11842b21e51dd330c0 P a3e62ccad691413ea2554182f73418a3: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:00.324244 31676 consensus_queue.cc:260] T 9051190ef3684094b383b9535e6aea83 P c0fad54e965647aaaa34b56674646cb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } }
I20240501 14:04:00.324446 31675 ts_tablet_manager.cc:1367] T 9051190ef3684094b383b9535e6aea83 P 99f6ca2d65d94b33b7a1d2341c24179a: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.004s
I20240501 14:04:00.324582 31677 tablet_bootstrap.cc:492] T e445777c9d99413a84d7546320cc621d P 8883498098f4454bbfe563fb83d31899: No bootstrap required, opened a new log
I20240501 14:04:00.324743 31677 ts_tablet_manager.cc:1367] T e445777c9d99413a84d7546320cc621d P 8883498098f4454bbfe563fb83d31899: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
I20240501 14:04:00.324820 31676 ts_tablet_manager.cc:1398] T 9051190ef3684094b383b9535e6aea83 P c0fad54e965647aaaa34b56674646cb4: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:04:00.325170 31675 raft_consensus.cc:348] T 9051190ef3684094b383b9535e6aea83 P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } }
I20240501 14:04:00.325524 31677 raft_consensus.cc:348] T e445777c9d99413a84d7546320cc621d P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:04:00.325651 31675 raft_consensus.cc:374] T 9051190ef3684094b383b9535e6aea83 P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:00.325728 31675 raft_consensus.cc:729] T 9051190ef3684094b383b9535e6aea83 P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99f6ca2d65d94b33b7a1d2341c24179a, State: Initialized, Role: FOLLOWER
I20240501 14:04:00.325753 31677 raft_consensus.cc:374] T e445777c9d99413a84d7546320cc621d P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:00.326014 31675 consensus_queue.cc:260] T 9051190ef3684094b383b9535e6aea83 P 99f6ca2d65d94b33b7a1d2341c24179a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } }
I20240501 14:04:00.325838 31677 raft_consensus.cc:729] T e445777c9d99413a84d7546320cc621d P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8883498098f4454bbfe563fb83d31899, State: Initialized, Role: FOLLOWER
I20240501 14:04:00.326568 31675 ts_tablet_manager.cc:1398] T 9051190ef3684094b383b9535e6aea83 P 99f6ca2d65d94b33b7a1d2341c24179a: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:04:00.326468 31677 consensus_queue.cc:260] T e445777c9d99413a84d7546320cc621d P 8883498098f4454bbfe563fb83d31899 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:04:00.327193 31677 ts_tablet_manager.cc:1398] T e445777c9d99413a84d7546320cc621d P 8883498098f4454bbfe563fb83d31899: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:04:00.330509 31678 tablet_bootstrap.cc:492] T 8db48542aec94b11842b21e51dd330c0 P a3e62ccad691413ea2554182f73418a3: No bootstrap required, opened a new log
I20240501 14:04:00.330670 31678 ts_tablet_manager.cc:1367] T 8db48542aec94b11842b21e51dd330c0 P a3e62ccad691413ea2554182f73418a3: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.003s
I20240501 14:04:00.331423 31678 raft_consensus.cc:348] T 8db48542aec94b11842b21e51dd330c0 P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:04:00.331637 31678 raft_consensus.cc:374] T 8db48542aec94b11842b21e51dd330c0 P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:00.331710 31678 raft_consensus.cc:729] T 8db48542aec94b11842b21e51dd330c0 P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3e62ccad691413ea2554182f73418a3, State: Initialized, Role: FOLLOWER
I20240501 14:04:00.331933 31678 consensus_queue.cc:260] T 8db48542aec94b11842b21e51dd330c0 P a3e62ccad691413ea2554182f73418a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:04:00.332410 31678 ts_tablet_manager.cc:1398] T 8db48542aec94b11842b21e51dd330c0 P a3e62ccad691413ea2554182f73418a3: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:04:00.378202 31656 raft_consensus.cc:482] T e445777c9d99413a84d7546320cc621d P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:04:00.378360 31656 raft_consensus.cc:504] T e445777c9d99413a84d7546320cc621d P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:04:00.379022 31656 leader_election.cc:288] T e445777c9d99413a84d7546320cc621d P 8883498098f4454bbfe563fb83d31899 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 870fbdcfdac84bacae0ad22dba25f626 (127.29.124.132:44267), 9964c1c76dd14f7dbce766b0c022ab51 (127.29.124.134:38101)
I20240501 14:04:00.379578 31032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e445777c9d99413a84d7546320cc621d" candidate_uuid: "8883498098f4454bbfe563fb83d31899" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9964c1c76dd14f7dbce766b0c022ab51" is_pre_election: true
I20240501 14:04:00.379578 30770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e445777c9d99413a84d7546320cc621d" candidate_uuid: "8883498098f4454bbfe563fb83d31899" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "870fbdcfdac84bacae0ad22dba25f626" is_pre_election: true
I20240501 14:04:00.379806 30770 raft_consensus.cc:2454] T e445777c9d99413a84d7546320cc621d P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8883498098f4454bbfe563fb83d31899 in term 0.
I20240501 14:04:00.379806 31032 raft_consensus.cc:2454] T e445777c9d99413a84d7546320cc621d P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8883498098f4454bbfe563fb83d31899 in term 0.
I20240501 14:04:00.380247 30445 leader_election.cc:302] T e445777c9d99413a84d7546320cc621d P 8883498098f4454bbfe563fb83d31899 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 870fbdcfdac84bacae0ad22dba25f626, 8883498098f4454bbfe563fb83d31899; no voters:
I20240501 14:04:00.380501 31656 raft_consensus.cc:2781] T e445777c9d99413a84d7546320cc621d P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:04:00.380625 31656 raft_consensus.cc:482] T e445777c9d99413a84d7546320cc621d P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:04:00.380714 31656 raft_consensus.cc:3037] T e445777c9d99413a84d7546320cc621d P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:00.382519 31656 raft_consensus.cc:504] T e445777c9d99413a84d7546320cc621d P 8883498098f4454bbfe563fb83d31899 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:04:00.383047 31656 leader_election.cc:288] T e445777c9d99413a84d7546320cc621d P 8883498098f4454bbfe563fb83d31899 [CANDIDATE]: Term 1 election: Requested vote from peers 870fbdcfdac84bacae0ad22dba25f626 (127.29.124.132:44267), 9964c1c76dd14f7dbce766b0c022ab51 (127.29.124.134:38101)
I20240501 14:04:00.383507 30770 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e445777c9d99413a84d7546320cc621d" candidate_uuid: "8883498098f4454bbfe563fb83d31899" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "870fbdcfdac84bacae0ad22dba25f626"
I20240501 14:04:00.383507 31032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e445777c9d99413a84d7546320cc621d" candidate_uuid: "8883498098f4454bbfe563fb83d31899" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9964c1c76dd14f7dbce766b0c022ab51"
I20240501 14:04:00.383697 31032 raft_consensus.cc:3037] T e445777c9d99413a84d7546320cc621d P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:00.383697 30770 raft_consensus.cc:3037] T e445777c9d99413a84d7546320cc621d P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:00.386444 30770 raft_consensus.cc:2454] T e445777c9d99413a84d7546320cc621d P 870fbdcfdac84bacae0ad22dba25f626 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8883498098f4454bbfe563fb83d31899 in term 1.
I20240501 14:04:00.386444 31032 raft_consensus.cc:2454] T e445777c9d99413a84d7546320cc621d P 9964c1c76dd14f7dbce766b0c022ab51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8883498098f4454bbfe563fb83d31899 in term 1.
I20240501 14:04:00.386847 30445 leader_election.cc:302] T e445777c9d99413a84d7546320cc621d P 8883498098f4454bbfe563fb83d31899 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 870fbdcfdac84bacae0ad22dba25f626, 8883498098f4454bbfe563fb83d31899; no voters:
I20240501 14:04:00.387193 31656 raft_consensus.cc:2781] T e445777c9d99413a84d7546320cc621d P 8883498098f4454bbfe563fb83d31899 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:04:00.387349 31656 raft_consensus.cc:686] T e445777c9d99413a84d7546320cc621d P 8883498098f4454bbfe563fb83d31899 [term 1 LEADER]: Becoming Leader. State: Replica: 8883498098f4454bbfe563fb83d31899, State: Running, Role: LEADER
I20240501 14:04:00.387662 31656 consensus_queue.cc:237] T e445777c9d99413a84d7546320cc621d P 8883498098f4454bbfe563fb83d31899 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:04:00.390216 30232 catalog_manager.cc:5484] T e445777c9d99413a84d7546320cc621d P 8883498098f4454bbfe563fb83d31899 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8883498098f4454bbfe563fb83d31899 (127.29.124.130). New cstate: current_term: 1 leader_uuid: "8883498098f4454bbfe563fb83d31899" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } health_report { overall_health: HEALTHY } } }
I20240501 14:04:00.478017 31114 raft_consensus.cc:482] T 8db48542aec94b11842b21e51dd330c0 P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:04:00.478220 31114 raft_consensus.cc:504] T 8db48542aec94b11842b21e51dd330c0 P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:04:00.479015 31114 leader_election.cc:288] T 8db48542aec94b11842b21e51dd330c0 P a3e62ccad691413ea2554182f73418a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0fad54e965647aaaa34b56674646cb4 (127.29.124.129:42363), 99f6ca2d65d94b33b7a1d2341c24179a (127.29.124.133:35195)
I20240501 14:04:00.479465 30378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db48542aec94b11842b21e51dd330c0" candidate_uuid: "a3e62ccad691413ea2554182f73418a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0fad54e965647aaaa34b56674646cb4" is_pre_election: true
I20240501 14:04:00.479714 30378 raft_consensus.cc:2454] T 8db48542aec94b11842b21e51dd330c0 P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3e62ccad691413ea2554182f73418a3 in term 0.
I20240501 14:04:00.480147 30574 leader_election.cc:302] T 8db48542aec94b11842b21e51dd330c0 P a3e62ccad691413ea2554182f73418a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3e62ccad691413ea2554182f73418a3, c0fad54e965647aaaa34b56674646cb4; no voters:
I20240501 14:04:00.480443 31114 raft_consensus.cc:2781] T 8db48542aec94b11842b21e51dd330c0 P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:04:00.480577 31114 raft_consensus.cc:482] T 8db48542aec94b11842b21e51dd330c0 P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:04:00.480705 31114 raft_consensus.cc:3037] T 8db48542aec94b11842b21e51dd330c0 P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:00.483258 31114 raft_consensus.cc:504] T 8db48542aec94b11842b21e51dd330c0 P a3e62ccad691413ea2554182f73418a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:04:00.484114 31114 leader_election.cc:288] T 8db48542aec94b11842b21e51dd330c0 P a3e62ccad691413ea2554182f73418a3 [CANDIDATE]: Term 1 election: Requested vote from peers c0fad54e965647aaaa34b56674646cb4 (127.29.124.129:42363), 99f6ca2d65d94b33b7a1d2341c24179a (127.29.124.133:35195)
I20240501 14:04:00.484342 30378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db48542aec94b11842b21e51dd330c0" candidate_uuid: "a3e62ccad691413ea2554182f73418a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0fad54e965647aaaa34b56674646cb4"
I20240501 14:04:00.484544 30378 raft_consensus.cc:3037] T 8db48542aec94b11842b21e51dd330c0 P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:00.486466 30901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db48542aec94b11842b21e51dd330c0" candidate_uuid: "a3e62ccad691413ea2554182f73418a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" is_pre_election: true
I20240501 14:04:00.486503 30902 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8db48542aec94b11842b21e51dd330c0" candidate_uuid: "a3e62ccad691413ea2554182f73418a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99f6ca2d65d94b33b7a1d2341c24179a"
I20240501 14:04:00.486711 30901 raft_consensus.cc:2454] T 8db48542aec94b11842b21e51dd330c0 P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3e62ccad691413ea2554182f73418a3 in term 0.
I20240501 14:04:00.487072 30378 raft_consensus.cc:2454] T 8db48542aec94b11842b21e51dd330c0 P c0fad54e965647aaaa34b56674646cb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3e62ccad691413ea2554182f73418a3 in term 1.
I20240501 14:04:00.487399 30574 leader_election.cc:302] T 8db48542aec94b11842b21e51dd330c0 P a3e62ccad691413ea2554182f73418a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a3e62ccad691413ea2554182f73418a3, c0fad54e965647aaaa34b56674646cb4; no voters: 99f6ca2d65d94b33b7a1d2341c24179a
I20240501 14:04:00.487655 31114 raft_consensus.cc:2781] T 8db48542aec94b11842b21e51dd330c0 P a3e62ccad691413ea2554182f73418a3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:04:00.488031 31114 raft_consensus.cc:686] T 8db48542aec94b11842b21e51dd330c0 P a3e62ccad691413ea2554182f73418a3 [term 1 LEADER]: Becoming Leader. State: Replica: a3e62ccad691413ea2554182f73418a3, State: Running, Role: LEADER
I20240501 14:04:00.488292 31114 consensus_queue.cc:237] T 8db48542aec94b11842b21e51dd330c0 P a3e62ccad691413ea2554182f73418a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:04:00.490713 30232 catalog_manager.cc:5484] T 8db48542aec94b11842b21e51dd330c0 P a3e62ccad691413ea2554182f73418a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to a3e62ccad691413ea2554182f73418a3 (127.29.124.131). New cstate: current_term: 1 leader_uuid: "a3e62ccad691413ea2554182f73418a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } health_report { overall_health: HEALTHY } } }
I20240501 14:04:00.516809 31596 raft_consensus.cc:482] T 9051190ef3684094b383b9535e6aea83 P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:04:00.516975 31596 raft_consensus.cc:504] T 9051190ef3684094b383b9535e6aea83 P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } }
I20240501 14:04:00.517645 31596 leader_election.cc:288] T 9051190ef3684094b383b9535e6aea83 P c0fad54e965647aaaa34b56674646cb4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3e62ccad691413ea2554182f73418a3 (127.29.124.131:43965), 99f6ca2d65d94b33b7a1d2341c24179a (127.29.124.133:35195)
I20240501 14:04:00.518126 30901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9051190ef3684094b383b9535e6aea83" candidate_uuid: "c0fad54e965647aaaa34b56674646cb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" is_pre_election: true
I20240501 14:04:00.518137 30639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9051190ef3684094b383b9535e6aea83" candidate_uuid: "c0fad54e965647aaaa34b56674646cb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3e62ccad691413ea2554182f73418a3" is_pre_election: true
I20240501 14:04:00.518389 30639 raft_consensus.cc:2454] T 9051190ef3684094b383b9535e6aea83 P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0fad54e965647aaaa34b56674646cb4 in term 0.
I20240501 14:04:00.518390 30901 raft_consensus.cc:2454] T 9051190ef3684094b383b9535e6aea83 P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0fad54e965647aaaa34b56674646cb4 in term 0.
I20240501 14:04:00.518806 30311 leader_election.cc:302] T 9051190ef3684094b383b9535e6aea83 P c0fad54e965647aaaa34b56674646cb4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 99f6ca2d65d94b33b7a1d2341c24179a, a3e62ccad691413ea2554182f73418a3, c0fad54e965647aaaa34b56674646cb4; no voters:
I20240501 14:04:00.519073 31596 raft_consensus.cc:2781] T 9051190ef3684094b383b9535e6aea83 P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:04:00.519193 31596 raft_consensus.cc:482] T 9051190ef3684094b383b9535e6aea83 P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:04:00.519287 31596 raft_consensus.cc:3037] T 9051190ef3684094b383b9535e6aea83 P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:00.521163 31596 raft_consensus.cc:504] T 9051190ef3684094b383b9535e6aea83 P c0fad54e965647aaaa34b56674646cb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } }
I20240501 14:04:00.521788 31596 leader_election.cc:288] T 9051190ef3684094b383b9535e6aea83 P c0fad54e965647aaaa34b56674646cb4 [CANDIDATE]: Term 1 election: Requested vote from peers a3e62ccad691413ea2554182f73418a3 (127.29.124.131:43965), 99f6ca2d65d94b33b7a1d2341c24179a (127.29.124.133:35195)
I20240501 14:04:00.522141 30639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9051190ef3684094b383b9535e6aea83" candidate_uuid: "c0fad54e965647aaaa34b56674646cb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3e62ccad691413ea2554182f73418a3"
I20240501 14:04:00.522270 30901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9051190ef3684094b383b9535e6aea83" candidate_uuid: "c0fad54e965647aaaa34b56674646cb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99f6ca2d65d94b33b7a1d2341c24179a"
I20240501 14:04:00.522358 30639 raft_consensus.cc:3037] T 9051190ef3684094b383b9535e6aea83 P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:00.522480 30901 raft_consensus.cc:3037] T 9051190ef3684094b383b9535e6aea83 P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:00.524758 30901 raft_consensus.cc:2454] T 9051190ef3684094b383b9535e6aea83 P 99f6ca2d65d94b33b7a1d2341c24179a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0fad54e965647aaaa34b56674646cb4 in term 1.
I20240501 14:04:00.524899 30639 raft_consensus.cc:2454] T 9051190ef3684094b383b9535e6aea83 P a3e62ccad691413ea2554182f73418a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0fad54e965647aaaa34b56674646cb4 in term 1.
I20240501 14:04:00.525115 30313 leader_election.cc:302] T 9051190ef3684094b383b9535e6aea83 P c0fad54e965647aaaa34b56674646cb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99f6ca2d65d94b33b7a1d2341c24179a, c0fad54e965647aaaa34b56674646cb4; no voters:
I20240501 14:04:00.525406 31596 raft_consensus.cc:2781] T 9051190ef3684094b383b9535e6aea83 P c0fad54e965647aaaa34b56674646cb4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:04:00.525514 31596 raft_consensus.cc:686] T 9051190ef3684094b383b9535e6aea83 P c0fad54e965647aaaa34b56674646cb4 [term 1 LEADER]: Becoming Leader. State: Replica: c0fad54e965647aaaa34b56674646cb4, State: Running, Role: LEADER
I20240501 14:04:00.525828 31596 consensus_queue.cc:237] T 9051190ef3684094b383b9535e6aea83 P c0fad54e965647aaaa34b56674646cb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } }
I20240501 14:04:00.528544 30232 catalog_manager.cc:5484] T 9051190ef3684094b383b9535e6aea83 P c0fad54e965647aaaa34b56674646cb4 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0fad54e965647aaaa34b56674646cb4 (127.29.124.129). New cstate: current_term: 1 leader_uuid: "c0fad54e965647aaaa34b56674646cb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } health_report { overall_health: UNKNOWN } } }
I20240501 14:04:00.886588 30901 raft_consensus.cc:3037] T 8db48542aec94b11842b21e51dd330c0 P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:00.889045 31114 consensus_queue.cc:1035] T 8db48542aec94b11842b21e51dd330c0 P a3e62ccad691413ea2554182f73418a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:04:00.895449 31693 consensus_queue.cc:1035] T 8db48542aec94b11842b21e51dd330c0 P a3e62ccad691413ea2554182f73418a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:04:00.910288 31696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:04:00.910621 31696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:04:00.918038 31696 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:04:00.927752 31596 consensus_queue.cc:1035] T 9051190ef3684094b383b9535e6aea83 P c0fad54e965647aaaa34b56674646cb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:04:00.932895 31611 consensus_queue.cc:1035] T 9051190ef3684094b383b9535e6aea83 P c0fad54e965647aaaa34b56674646cb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:04:00.940313 31656 consensus_queue.cc:1035] T e445777c9d99413a84d7546320cc621d P 8883498098f4454bbfe563fb83d31899 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:04:00.967486 31656 consensus_queue.cc:1035] T e445777c9d99413a84d7546320cc621d P 8883498098f4454bbfe563fb83d31899 [LEADER]: Connected to new peer: Peer: permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_6'
INSERT report
rows total: 2000
time total: 246.383 ms
time per row: 0.123191 ms
I20240501 14:04:01.344728 30232 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35960:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:04:01.354483 30882 tablet_service.cc:1467] Processing CreateTablet for tablet 1f5ff3b1c34a4de780fe4783b9b0eaf1 (DEFAULT_TABLE table=rebalance_test_table_7 [id=c307346082bb42728f205da4acf38cba]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:01.354599 30357 tablet_service.cc:1467] Processing CreateTablet for tablet 1f5ff3b1c34a4de780fe4783b9b0eaf1 (DEFAULT_TABLE table=rebalance_test_table_7 [id=c307346082bb42728f205da4acf38cba]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:01.354771 30489 tablet_service.cc:1467] Processing CreateTablet for tablet 4b44fc1c4c33499a90f6c446110d9c42 (DEFAULT_TABLE table=rebalance_test_table_7 [id=c307346082bb42728f205da4acf38cba]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:01.354959 30882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f5ff3b1c34a4de780fe4783b9b0eaf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:01.355037 30357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f5ff3b1c34a4de780fe4783b9b0eaf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:01.355123 30489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b44fc1c4c33499a90f6c446110d9c42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:01.355078 30750 tablet_service.cc:1467] Processing CreateTablet for tablet 4b44fc1c4c33499a90f6c446110d9c42 (DEFAULT_TABLE table=rebalance_test_table_7 [id=c307346082bb42728f205da4acf38cba]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:01.355468 30750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b44fc1c4c33499a90f6c446110d9c42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:01.361449 30751 tablet_service.cc:1467] Processing CreateTablet for tablet 51ed183d7ec2459faf1b25232b3e504d (DEFAULT_TABLE table=rebalance_test_table_7 [id=c307346082bb42728f205da4acf38cba]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:01.361727 31012 tablet_service.cc:1467] Processing CreateTablet for tablet 51ed183d7ec2459faf1b25232b3e504d (DEFAULT_TABLE table=rebalance_test_table_7 [id=c307346082bb42728f205da4acf38cba]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:01.362157 31012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51ed183d7ec2459faf1b25232b3e504d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:01.363468 30620 tablet_service.cc:1467] Processing CreateTablet for tablet 1f5ff3b1c34a4de780fe4783b9b0eaf1 (DEFAULT_TABLE table=rebalance_test_table_7 [id=c307346082bb42728f205da4acf38cba]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:01.363871 30620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f5ff3b1c34a4de780fe4783b9b0eaf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:01.365638 31768 tablet_bootstrap.cc:492] T 1f5ff3b1c34a4de780fe4783b9b0eaf1 P c0fad54e965647aaaa34b56674646cb4: Bootstrap starting.
I20240501 14:04:01.365595 31013 tablet_service.cc:1467] Processing CreateTablet for tablet 4b44fc1c4c33499a90f6c446110d9c42 (DEFAULT_TABLE table=rebalance_test_table_7 [id=c307346082bb42728f205da4acf38cba]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:01.365931 31013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4b44fc1c4c33499a90f6c446110d9c42. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:01.368619 31768 tablet_bootstrap.cc:654] T 1f5ff3b1c34a4de780fe4783b9b0eaf1 P c0fad54e965647aaaa34b56674646cb4: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:01.369880 30488 tablet_service.cc:1467] Processing CreateTablet for tablet 51ed183d7ec2459faf1b25232b3e504d (DEFAULT_TABLE table=rebalance_test_table_7 [id=c307346082bb42728f205da4acf38cba]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:01.370162 31767 tablet_bootstrap.cc:492] T 4b44fc1c4c33499a90f6c446110d9c42 P 8883498098f4454bbfe563fb83d31899: Bootstrap starting.
I20240501 14:04:01.370322 30488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51ed183d7ec2459faf1b25232b3e504d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:01.372354 31767 tablet_bootstrap.cc:654] T 4b44fc1c4c33499a90f6c446110d9c42 P 8883498098f4454bbfe563fb83d31899: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:01.381044 30751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51ed183d7ec2459faf1b25232b3e504d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:01.382565 31770 tablet_bootstrap.cc:492] T 1f5ff3b1c34a4de780fe4783b9b0eaf1 P a3e62ccad691413ea2554182f73418a3: Bootstrap starting.
I20240501 14:04:01.388440 31770 tablet_bootstrap.cc:654] T 1f5ff3b1c34a4de780fe4783b9b0eaf1 P a3e62ccad691413ea2554182f73418a3: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:01.388976 31772 tablet_bootstrap.cc:492] T 4b44fc1c4c33499a90f6c446110d9c42 P 870fbdcfdac84bacae0ad22dba25f626: Bootstrap starting.
I20240501 14:04:01.390798 31766 tablet_bootstrap.cc:492] T 1f5ff3b1c34a4de780fe4783b9b0eaf1 P 99f6ca2d65d94b33b7a1d2341c24179a: Bootstrap starting.
I20240501 14:04:01.391010 31772 tablet_bootstrap.cc:654] T 4b44fc1c4c33499a90f6c446110d9c42 P 870fbdcfdac84bacae0ad22dba25f626: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:01.392733 31766 tablet_bootstrap.cc:654] T 1f5ff3b1c34a4de780fe4783b9b0eaf1 P 99f6ca2d65d94b33b7a1d2341c24179a: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:01.393553 31771 tablet_bootstrap.cc:492] T 51ed183d7ec2459faf1b25232b3e504d P 9964c1c76dd14f7dbce766b0c022ab51: Bootstrap starting.
I20240501 14:04:01.394870 31771 tablet_bootstrap.cc:654] T 51ed183d7ec2459faf1b25232b3e504d P 9964c1c76dd14f7dbce766b0c022ab51: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:01.396312 31772 tablet_bootstrap.cc:492] T 4b44fc1c4c33499a90f6c446110d9c42 P 870fbdcfdac84bacae0ad22dba25f626: No bootstrap required, opened a new log
I20240501 14:04:01.396464 31772 ts_tablet_manager.cc:1367] T 4b44fc1c4c33499a90f6c446110d9c42 P 870fbdcfdac84bacae0ad22dba25f626: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240501 14:04:01.398746 31772 raft_consensus.cc:348] T 4b44fc1c4c33499a90f6c446110d9c42 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:04:01.399755 31768 tablet_bootstrap.cc:492] T 1f5ff3b1c34a4de780fe4783b9b0eaf1 P c0fad54e965647aaaa34b56674646cb4: No bootstrap required, opened a new log
I20240501 14:04:01.399974 31768 ts_tablet_manager.cc:1367] T 1f5ff3b1c34a4de780fe4783b9b0eaf1 P c0fad54e965647aaaa34b56674646cb4: Time spent bootstrapping tablet: real 0.034s user 0.005s sys 0.011s
I20240501 14:04:01.401682 31767 tablet_bootstrap.cc:492] T 4b44fc1c4c33499a90f6c446110d9c42 P 8883498098f4454bbfe563fb83d31899: No bootstrap required, opened a new log
I20240501 14:04:01.401831 31767 ts_tablet_manager.cc:1367] T 4b44fc1c4c33499a90f6c446110d9c42 P 8883498098f4454bbfe563fb83d31899: Time spent bootstrapping tablet: real 0.032s user 0.011s sys 0.000s
I20240501 14:04:01.401854 31770 tablet_bootstrap.cc:492] T 1f5ff3b1c34a4de780fe4783b9b0eaf1 P a3e62ccad691413ea2554182f73418a3: No bootstrap required, opened a new log
I20240501 14:04:01.401687 31768 raft_consensus.cc:348] T 1f5ff3b1c34a4de780fe4783b9b0eaf1 P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:04:01.401993 31770 ts_tablet_manager.cc:1367] T 1f5ff3b1c34a4de780fe4783b9b0eaf1 P a3e62ccad691413ea2554182f73418a3: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.000s
I20240501 14:04:01.402119 31768 raft_consensus.cc:374] T 1f5ff3b1c34a4de780fe4783b9b0eaf1 P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:01.402261 31768 raft_consensus.cc:729] T 1f5ff3b1c34a4de780fe4783b9b0eaf1 P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0fad54e965647aaaa34b56674646cb4, State: Initialized, Role: FOLLOWER
I20240501 14:04:01.402530 31767 raft_consensus.cc:348] T 4b44fc1c4c33499a90f6c446110d9c42 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:04:01.402725 31767 raft_consensus.cc:374] T 4b44fc1c4c33499a90f6c446110d9c42 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:01.402801 31767 raft_consensus.cc:729] T 4b44fc1c4c33499a90f6c446110d9c42 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8883498098f4454bbfe563fb83d31899, State: Initialized, Role: FOLLOWER
I20240501 14:04:01.402617 31768 consensus_queue.cc:260] T 1f5ff3b1c34a4de780fe4783b9b0eaf1 P c0fad54e965647aaaa34b56674646cb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:04:01.402935 31770 raft_consensus.cc:348] T 1f5ff3b1c34a4de780fe4783b9b0eaf1 P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:04:01.402984 31767 consensus_queue.cc:260] T 4b44fc1c4c33499a90f6c446110d9c42 P 8883498098f4454bbfe563fb83d31899 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:04:01.403131 31770 raft_consensus.cc:374] T 1f5ff3b1c34a4de780fe4783b9b0eaf1 P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:01.403349 31770 raft_consensus.cc:729] T 1f5ff3b1c34a4de780fe4783b9b0eaf1 P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3e62ccad691413ea2554182f73418a3, State: Initialized, Role: FOLLOWER
I20240501 14:04:01.403617 31770 consensus_queue.cc:260] T 1f5ff3b1c34a4de780fe4783b9b0eaf1 P a3e62ccad691413ea2554182f73418a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:04:01.404402 31767 ts_tablet_manager.cc:1398] T 4b44fc1c4c33499a90f6c446110d9c42 P 8883498098f4454bbfe563fb83d31899: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:04:01.404639 31767 tablet_bootstrap.cc:492] T 51ed183d7ec2459faf1b25232b3e504d P 8883498098f4454bbfe563fb83d31899: Bootstrap starting.
I20240501 14:04:01.406179 31767 tablet_bootstrap.cc:654] T 51ed183d7ec2459faf1b25232b3e504d P 8883498098f4454bbfe563fb83d31899: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:01.406638 31768 ts_tablet_manager.cc:1398] T 1f5ff3b1c34a4de780fe4783b9b0eaf1 P c0fad54e965647aaaa34b56674646cb4: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:04:01.406908 31772 raft_consensus.cc:374] T 4b44fc1c4c33499a90f6c446110d9c42 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:01.407109 31772 raft_consensus.cc:729] T 4b44fc1c4c33499a90f6c446110d9c42 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 870fbdcfdac84bacae0ad22dba25f626, State: Initialized, Role: FOLLOWER
I20240501 14:04:01.407516 31772 consensus_queue.cc:260] T 4b44fc1c4c33499a90f6c446110d9c42 P 870fbdcfdac84bacae0ad22dba25f626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:04:01.412492 31770 ts_tablet_manager.cc:1398] T 1f5ff3b1c34a4de780fe4783b9b0eaf1 P a3e62ccad691413ea2554182f73418a3: Time spent starting tablet: real 0.010s user 0.000s sys 0.003s
I20240501 14:04:01.412787 31772 ts_tablet_manager.cc:1398] T 4b44fc1c4c33499a90f6c446110d9c42 P 870fbdcfdac84bacae0ad22dba25f626: Time spent starting tablet: real 0.016s user 0.000s sys 0.003s
I20240501 14:04:01.414245 31766 tablet_bootstrap.cc:492] T 1f5ff3b1c34a4de780fe4783b9b0eaf1 P 99f6ca2d65d94b33b7a1d2341c24179a: No bootstrap required, opened a new log
I20240501 14:04:01.414425 31766 ts_tablet_manager.cc:1367] T 1f5ff3b1c34a4de780fe4783b9b0eaf1 P 99f6ca2d65d94b33b7a1d2341c24179a: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.000s
I20240501 14:04:01.415378 31772 tablet_bootstrap.cc:492] T 51ed183d7ec2459faf1b25232b3e504d P 870fbdcfdac84bacae0ad22dba25f626: Bootstrap starting.
I20240501 14:04:01.416455 31771 tablet_bootstrap.cc:492] T 51ed183d7ec2459faf1b25232b3e504d P 9964c1c76dd14f7dbce766b0c022ab51: No bootstrap required, opened a new log
I20240501 14:04:01.416637 31772 tablet_bootstrap.cc:654] T 51ed183d7ec2459faf1b25232b3e504d P 870fbdcfdac84bacae0ad22dba25f626: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:01.416628 31771 ts_tablet_manager.cc:1367] T 51ed183d7ec2459faf1b25232b3e504d P 9964c1c76dd14f7dbce766b0c022ab51: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.000s
I20240501 14:04:01.417418 31771 raft_consensus.cc:348] T 51ed183d7ec2459faf1b25232b3e504d P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:04:01.417621 31771 raft_consensus.cc:374] T 51ed183d7ec2459faf1b25232b3e504d P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:01.417697 31771 raft_consensus.cc:729] T 51ed183d7ec2459faf1b25232b3e504d P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9964c1c76dd14f7dbce766b0c022ab51, State: Initialized, Role: FOLLOWER
I20240501 14:04:01.418105 31766 raft_consensus.cc:348] T 1f5ff3b1c34a4de780fe4783b9b0eaf1 P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:04:01.418319 31766 raft_consensus.cc:374] T 1f5ff3b1c34a4de780fe4783b9b0eaf1 P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:01.418359 31772 tablet_bootstrap.cc:492] T 51ed183d7ec2459faf1b25232b3e504d P 870fbdcfdac84bacae0ad22dba25f626: No bootstrap required, opened a new log
I20240501 14:04:01.418519 31772 ts_tablet_manager.cc:1367] T 51ed183d7ec2459faf1b25232b3e504d P 870fbdcfdac84bacae0ad22dba25f626: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.003s
I20240501 14:04:01.418557 31771 consensus_queue.cc:260] T 51ed183d7ec2459faf1b25232b3e504d P 9964c1c76dd14f7dbce766b0c022ab51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:04:01.419219 31772 raft_consensus.cc:348] T 51ed183d7ec2459faf1b25232b3e504d P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:04:01.419406 31772 raft_consensus.cc:374] T 51ed183d7ec2459faf1b25232b3e504d P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:01.419477 31772 raft_consensus.cc:729] T 51ed183d7ec2459faf1b25232b3e504d P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 870fbdcfdac84bacae0ad22dba25f626, State: Initialized, Role: FOLLOWER
I20240501 14:04:01.419698 31772 consensus_queue.cc:260] T 51ed183d7ec2459faf1b25232b3e504d P 870fbdcfdac84bacae0ad22dba25f626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:04:01.420045 31771 ts_tablet_manager.cc:1398] T 51ed183d7ec2459faf1b25232b3e504d P 9964c1c76dd14f7dbce766b0c022ab51: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:04:01.420315 31771 tablet_bootstrap.cc:492] T 4b44fc1c4c33499a90f6c446110d9c42 P 9964c1c76dd14f7dbce766b0c022ab51: Bootstrap starting.
I20240501 14:04:01.420851 31766 raft_consensus.cc:729] T 1f5ff3b1c34a4de780fe4783b9b0eaf1 P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99f6ca2d65d94b33b7a1d2341c24179a, State: Initialized, Role: FOLLOWER
I20240501 14:04:01.421094 31766 consensus_queue.cc:260] T 1f5ff3b1c34a4de780fe4783b9b0eaf1 P 99f6ca2d65d94b33b7a1d2341c24179a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:04:01.422699 31766 ts_tablet_manager.cc:1398] T 1f5ff3b1c34a4de780fe4783b9b0eaf1 P 99f6ca2d65d94b33b7a1d2341c24179a: Time spent starting tablet: real 0.008s user 0.000s sys 0.003s
I20240501 14:04:01.423884 31771 tablet_bootstrap.cc:654] T 4b44fc1c4c33499a90f6c446110d9c42 P 9964c1c76dd14f7dbce766b0c022ab51: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:01.424681 31772 ts_tablet_manager.cc:1398] T 51ed183d7ec2459faf1b25232b3e504d P 870fbdcfdac84bacae0ad22dba25f626: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240501 14:04:01.424971 31767 tablet_bootstrap.cc:492] T 51ed183d7ec2459faf1b25232b3e504d P 8883498098f4454bbfe563fb83d31899: No bootstrap required, opened a new log
I20240501 14:04:01.425141 31767 ts_tablet_manager.cc:1367] T 51ed183d7ec2459faf1b25232b3e504d P 8883498098f4454bbfe563fb83d31899: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.000s
I20240501 14:04:01.425959 31767 raft_consensus.cc:348] T 51ed183d7ec2459faf1b25232b3e504d P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:04:01.426188 31767 raft_consensus.cc:374] T 51ed183d7ec2459faf1b25232b3e504d P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:01.426278 31767 raft_consensus.cc:729] T 51ed183d7ec2459faf1b25232b3e504d P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8883498098f4454bbfe563fb83d31899, State: Initialized, Role: FOLLOWER
I20240501 14:04:01.426522 31767 consensus_queue.cc:260] T 51ed183d7ec2459faf1b25232b3e504d P 8883498098f4454bbfe563fb83d31899 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:04:01.427060 31767 ts_tablet_manager.cc:1398] T 51ed183d7ec2459faf1b25232b3e504d P 8883498098f4454bbfe563fb83d31899: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:04:01.438216 31771 tablet_bootstrap.cc:492] T 4b44fc1c4c33499a90f6c446110d9c42 P 9964c1c76dd14f7dbce766b0c022ab51: No bootstrap required, opened a new log
I20240501 14:04:01.438413 31771 ts_tablet_manager.cc:1367] T 4b44fc1c4c33499a90f6c446110d9c42 P 9964c1c76dd14f7dbce766b0c022ab51: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.000s
I20240501 14:04:01.439153 31771 raft_consensus.cc:348] T 4b44fc1c4c33499a90f6c446110d9c42 P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:04:01.439379 31771 raft_consensus.cc:374] T 4b44fc1c4c33499a90f6c446110d9c42 P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:01.439448 31771 raft_consensus.cc:729] T 4b44fc1c4c33499a90f6c446110d9c42 P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9964c1c76dd14f7dbce766b0c022ab51, State: Initialized, Role: FOLLOWER
I20240501 14:04:01.439631 31771 consensus_queue.cc:260] T 4b44fc1c4c33499a90f6c446110d9c42 P 9964c1c76dd14f7dbce766b0c022ab51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:04:01.440078 31771 ts_tablet_manager.cc:1398] T 4b44fc1c4c33499a90f6c446110d9c42 P 9964c1c76dd14f7dbce766b0c022ab51: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:04:01.488719 31114 raft_consensus.cc:482] T 1f5ff3b1c34a4de780fe4783b9b0eaf1 P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:04:01.488930 31114 raft_consensus.cc:504] T 1f5ff3b1c34a4de780fe4783b9b0eaf1 P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:04:01.489625 31114 leader_election.cc:288] T 1f5ff3b1c34a4de780fe4783b9b0eaf1 P a3e62ccad691413ea2554182f73418a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0fad54e965647aaaa34b56674646cb4 (127.29.124.129:42363), 99f6ca2d65d94b33b7a1d2341c24179a (127.29.124.133:35195)
I20240501 14:04:01.490123 30378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5ff3b1c34a4de780fe4783b9b0eaf1" candidate_uuid: "a3e62ccad691413ea2554182f73418a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0fad54e965647aaaa34b56674646cb4" is_pre_election: true
I20240501 14:04:01.490171 30901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5ff3b1c34a4de780fe4783b9b0eaf1" candidate_uuid: "a3e62ccad691413ea2554182f73418a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" is_pre_election: true
I20240501 14:04:01.490353 30378 raft_consensus.cc:2454] T 1f5ff3b1c34a4de780fe4783b9b0eaf1 P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3e62ccad691413ea2554182f73418a3 in term 0.
I20240501 14:04:01.490401 30901 raft_consensus.cc:2454] T 1f5ff3b1c34a4de780fe4783b9b0eaf1 P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3e62ccad691413ea2554182f73418a3 in term 0.
I20240501 14:04:01.490779 30574 leader_election.cc:302] T 1f5ff3b1c34a4de780fe4783b9b0eaf1 P a3e62ccad691413ea2554182f73418a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3e62ccad691413ea2554182f73418a3, c0fad54e965647aaaa34b56674646cb4; no voters:
I20240501 14:04:01.491067 31114 raft_consensus.cc:2781] T 1f5ff3b1c34a4de780fe4783b9b0eaf1 P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:04:01.491164 31114 raft_consensus.cc:482] T 1f5ff3b1c34a4de780fe4783b9b0eaf1 P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:04:01.491267 31114 raft_consensus.cc:3037] T 1f5ff3b1c34a4de780fe4783b9b0eaf1 P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:01.493295 31114 raft_consensus.cc:504] T 1f5ff3b1c34a4de780fe4783b9b0eaf1 P a3e62ccad691413ea2554182f73418a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:04:01.493849 31114 leader_election.cc:288] T 1f5ff3b1c34a4de780fe4783b9b0eaf1 P a3e62ccad691413ea2554182f73418a3 [CANDIDATE]: Term 1 election: Requested vote from peers c0fad54e965647aaaa34b56674646cb4 (127.29.124.129:42363), 99f6ca2d65d94b33b7a1d2341c24179a (127.29.124.133:35195)
I20240501 14:04:01.494230 30378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5ff3b1c34a4de780fe4783b9b0eaf1" candidate_uuid: "a3e62ccad691413ea2554182f73418a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0fad54e965647aaaa34b56674646cb4"
I20240501 14:04:01.494288 30901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f5ff3b1c34a4de780fe4783b9b0eaf1" candidate_uuid: "a3e62ccad691413ea2554182f73418a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99f6ca2d65d94b33b7a1d2341c24179a"
I20240501 14:04:01.494410 30378 raft_consensus.cc:3037] T 1f5ff3b1c34a4de780fe4783b9b0eaf1 P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:01.494488 30901 raft_consensus.cc:3037] T 1f5ff3b1c34a4de780fe4783b9b0eaf1 P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:01.496265 30901 raft_consensus.cc:2454] T 1f5ff3b1c34a4de780fe4783b9b0eaf1 P 99f6ca2d65d94b33b7a1d2341c24179a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3e62ccad691413ea2554182f73418a3 in term 1.
I20240501 14:04:01.496500 30378 raft_consensus.cc:2454] T 1f5ff3b1c34a4de780fe4783b9b0eaf1 P c0fad54e965647aaaa34b56674646cb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3e62ccad691413ea2554182f73418a3 in term 1.
I20240501 14:04:01.496660 30575 leader_election.cc:302] T 1f5ff3b1c34a4de780fe4783b9b0eaf1 P a3e62ccad691413ea2554182f73418a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99f6ca2d65d94b33b7a1d2341c24179a, a3e62ccad691413ea2554182f73418a3; no voters:
I20240501 14:04:01.496886 31114 raft_consensus.cc:2781] T 1f5ff3b1c34a4de780fe4783b9b0eaf1 P a3e62ccad691413ea2554182f73418a3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:04:01.497047 31114 raft_consensus.cc:686] T 1f5ff3b1c34a4de780fe4783b9b0eaf1 P a3e62ccad691413ea2554182f73418a3 [term 1 LEADER]: Becoming Leader. State: Replica: a3e62ccad691413ea2554182f73418a3, State: Running, Role: LEADER
I20240501 14:04:01.497359 31114 consensus_queue.cc:237] T 1f5ff3b1c34a4de780fe4783b9b0eaf1 P a3e62ccad691413ea2554182f73418a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:04:01.500072 30232 catalog_manager.cc:5484] T 1f5ff3b1c34a4de780fe4783b9b0eaf1 P a3e62ccad691413ea2554182f73418a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to a3e62ccad691413ea2554182f73418a3 (127.29.124.131). New cstate: current_term: 1 leader_uuid: "a3e62ccad691413ea2554182f73418a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } health_report { overall_health: HEALTHY } } }
I20240501 14:04:01.516299 31709 raft_consensus.cc:482] T 4b44fc1c4c33499a90f6c446110d9c42 P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:04:01.516494 31709 raft_consensus.cc:504] T 4b44fc1c4c33499a90f6c446110d9c42 P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:04:01.518024 31709 leader_election.cc:288] T 4b44fc1c4c33499a90f6c446110d9c42 P 9964c1c76dd14f7dbce766b0c022ab51 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 870fbdcfdac84bacae0ad22dba25f626 (127.29.124.132:44267), 8883498098f4454bbfe563fb83d31899 (127.29.124.130:43099)
I20240501 14:04:01.521224 31199 raft_consensus.cc:482] T 51ed183d7ec2459faf1b25232b3e504d P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:04:01.521446 31199 raft_consensus.cc:504] T 51ed183d7ec2459faf1b25232b3e504d P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:04:01.522213 31199 leader_election.cc:288] T 51ed183d7ec2459faf1b25232b3e504d P 870fbdcfdac84bacae0ad22dba25f626 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9964c1c76dd14f7dbce766b0c022ab51 (127.29.124.134:38101), 8883498098f4454bbfe563fb83d31899 (127.29.124.130:43099)
I20240501 14:04:01.522617 31032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51ed183d7ec2459faf1b25232b3e504d" candidate_uuid: "870fbdcfdac84bacae0ad22dba25f626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9964c1c76dd14f7dbce766b0c022ab51" is_pre_election: true
I20240501 14:04:01.522893 31032 raft_consensus.cc:2454] T 51ed183d7ec2459faf1b25232b3e504d P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 870fbdcfdac84bacae0ad22dba25f626 in term 0.
I20240501 14:04:01.523360 30704 leader_election.cc:302] T 51ed183d7ec2459faf1b25232b3e504d P 870fbdcfdac84bacae0ad22dba25f626 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 870fbdcfdac84bacae0ad22dba25f626, 9964c1c76dd14f7dbce766b0c022ab51; no voters:
I20240501 14:04:01.523507 30509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51ed183d7ec2459faf1b25232b3e504d" candidate_uuid: "870fbdcfdac84bacae0ad22dba25f626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8883498098f4454bbfe563fb83d31899" is_pre_election: true
I20240501 14:04:01.523720 31199 raft_consensus.cc:2781] T 51ed183d7ec2459faf1b25232b3e504d P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:04:01.523730 30509 raft_consensus.cc:2454] T 51ed183d7ec2459faf1b25232b3e504d P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 870fbdcfdac84bacae0ad22dba25f626 in term 0.
I20240501 14:04:01.523824 31199 raft_consensus.cc:482] T 51ed183d7ec2459faf1b25232b3e504d P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:04:01.523907 31199 raft_consensus.cc:3037] T 51ed183d7ec2459faf1b25232b3e504d P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:01.525949 30509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b44fc1c4c33499a90f6c446110d9c42" candidate_uuid: "9964c1c76dd14f7dbce766b0c022ab51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8883498098f4454bbfe563fb83d31899" is_pre_election: true
I20240501 14:04:01.526154 30509 raft_consensus.cc:2454] T 4b44fc1c4c33499a90f6c446110d9c42 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9964c1c76dd14f7dbce766b0c022ab51 in term 0.
I20240501 14:04:01.526580 31199 raft_consensus.cc:504] T 51ed183d7ec2459faf1b25232b3e504d P 870fbdcfdac84bacae0ad22dba25f626 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:04:01.527024 30771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b44fc1c4c33499a90f6c446110d9c42" candidate_uuid: "9964c1c76dd14f7dbce766b0c022ab51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "870fbdcfdac84bacae0ad22dba25f626" is_pre_election: true
I20240501 14:04:01.527225 30771 raft_consensus.cc:2454] T 4b44fc1c4c33499a90f6c446110d9c42 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9964c1c76dd14f7dbce766b0c022ab51 in term 0.
I20240501 14:04:01.527320 30969 leader_election.cc:302] T 4b44fc1c4c33499a90f6c446110d9c42 P 9964c1c76dd14f7dbce766b0c022ab51 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8883498098f4454bbfe563fb83d31899, 9964c1c76dd14f7dbce766b0c022ab51; no voters:
I20240501 14:04:01.527702 31709 raft_consensus.cc:2781] T 4b44fc1c4c33499a90f6c446110d9c42 P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:04:01.527842 31709 raft_consensus.cc:482] T 4b44fc1c4c33499a90f6c446110d9c42 P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:04:01.527937 31709 raft_consensus.cc:3037] T 4b44fc1c4c33499a90f6c446110d9c42 P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:01.528381 31199 leader_election.cc:288] T 51ed183d7ec2459faf1b25232b3e504d P 870fbdcfdac84bacae0ad22dba25f626 [CANDIDATE]: Term 1 election: Requested vote from peers 9964c1c76dd14f7dbce766b0c022ab51 (127.29.124.134:38101), 8883498098f4454bbfe563fb83d31899 (127.29.124.130:43099)
I20240501 14:04:01.528914 30509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51ed183d7ec2459faf1b25232b3e504d" candidate_uuid: "870fbdcfdac84bacae0ad22dba25f626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8883498098f4454bbfe563fb83d31899"
I20240501 14:04:01.529094 30509 raft_consensus.cc:3037] T 51ed183d7ec2459faf1b25232b3e504d P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:01.529120 31032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51ed183d7ec2459faf1b25232b3e504d" candidate_uuid: "870fbdcfdac84bacae0ad22dba25f626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9964c1c76dd14f7dbce766b0c022ab51"
I20240501 14:04:01.529400 31032 raft_consensus.cc:3037] T 51ed183d7ec2459faf1b25232b3e504d P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:01.530637 31709 raft_consensus.cc:504] T 4b44fc1c4c33499a90f6c446110d9c42 P 9964c1c76dd14f7dbce766b0c022ab51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:04:01.531165 30509 raft_consensus.cc:2454] T 51ed183d7ec2459faf1b25232b3e504d P 8883498098f4454bbfe563fb83d31899 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 870fbdcfdac84bacae0ad22dba25f626 in term 1.
I20240501 14:04:01.531333 31709 leader_election.cc:288] T 4b44fc1c4c33499a90f6c446110d9c42 P 9964c1c76dd14f7dbce766b0c022ab51 [CANDIDATE]: Term 1 election: Requested vote from peers 870fbdcfdac84bacae0ad22dba25f626 (127.29.124.132:44267), 8883498098f4454bbfe563fb83d31899 (127.29.124.130:43099)
I20240501 14:04:01.531756 30707 leader_election.cc:302] T 51ed183d7ec2459faf1b25232b3e504d P 870fbdcfdac84bacae0ad22dba25f626 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 870fbdcfdac84bacae0ad22dba25f626, 8883498098f4454bbfe563fb83d31899; no voters:
I20240501 14:04:01.531884 30771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b44fc1c4c33499a90f6c446110d9c42" candidate_uuid: "9964c1c76dd14f7dbce766b0c022ab51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "870fbdcfdac84bacae0ad22dba25f626"
I20240501 14:04:01.532021 31199 raft_consensus.cc:2781] T 51ed183d7ec2459faf1b25232b3e504d P 870fbdcfdac84bacae0ad22dba25f626 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:04:01.532059 31032 raft_consensus.cc:2454] T 51ed183d7ec2459faf1b25232b3e504d P 9964c1c76dd14f7dbce766b0c022ab51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 870fbdcfdac84bacae0ad22dba25f626 in term 1.
I20240501 14:04:01.532097 30771 raft_consensus.cc:3037] T 4b44fc1c4c33499a90f6c446110d9c42 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:01.532114 30509 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4b44fc1c4c33499a90f6c446110d9c42" candidate_uuid: "9964c1c76dd14f7dbce766b0c022ab51" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8883498098f4454bbfe563fb83d31899"
I20240501 14:04:01.532301 30509 raft_consensus.cc:3037] T 4b44fc1c4c33499a90f6c446110d9c42 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:01.532177 31199 raft_consensus.cc:686] T 51ed183d7ec2459faf1b25232b3e504d P 870fbdcfdac84bacae0ad22dba25f626 [term 1 LEADER]: Becoming Leader. State: Replica: 870fbdcfdac84bacae0ad22dba25f626, State: Running, Role: LEADER
I20240501 14:04:01.532656 31199 consensus_queue.cc:237] T 51ed183d7ec2459faf1b25232b3e504d P 870fbdcfdac84bacae0ad22dba25f626 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:04:01.534446 30771 raft_consensus.cc:2454] T 4b44fc1c4c33499a90f6c446110d9c42 P 870fbdcfdac84bacae0ad22dba25f626 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9964c1c76dd14f7dbce766b0c022ab51 in term 1.
I20240501 14:04:01.534737 30509 raft_consensus.cc:2454] T 4b44fc1c4c33499a90f6c446110d9c42 P 8883498098f4454bbfe563fb83d31899 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9964c1c76dd14f7dbce766b0c022ab51 in term 1.
I20240501 14:04:01.535013 30969 leader_election.cc:302] T 4b44fc1c4c33499a90f6c446110d9c42 P 9964c1c76dd14f7dbce766b0c022ab51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 870fbdcfdac84bacae0ad22dba25f626, 9964c1c76dd14f7dbce766b0c022ab51; no voters:
I20240501 14:04:01.535288 31709 raft_consensus.cc:2781] T 4b44fc1c4c33499a90f6c446110d9c42 P 9964c1c76dd14f7dbce766b0c022ab51 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:04:01.535686 31709 raft_consensus.cc:686] T 4b44fc1c4c33499a90f6c446110d9c42 P 9964c1c76dd14f7dbce766b0c022ab51 [term 1 LEADER]: Becoming Leader. State: Replica: 9964c1c76dd14f7dbce766b0c022ab51, State: Running, Role: LEADER
I20240501 14:04:01.536159 31709 consensus_queue.cc:237] T 4b44fc1c4c33499a90f6c446110d9c42 P 9964c1c76dd14f7dbce766b0c022ab51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:04:01.536103 30232 catalog_manager.cc:5484] T 51ed183d7ec2459faf1b25232b3e504d P 870fbdcfdac84bacae0ad22dba25f626 reported cstate change: term changed from 0 to 1, leader changed from <none> to 870fbdcfdac84bacae0ad22dba25f626 (127.29.124.132). New cstate: current_term: 1 leader_uuid: "870fbdcfdac84bacae0ad22dba25f626" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } health_report { overall_health: UNKNOWN } } }
I20240501 14:04:01.539659 30232 catalog_manager.cc:5484] T 4b44fc1c4c33499a90f6c446110d9c42 P 9964c1c76dd14f7dbce766b0c022ab51 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9964c1c76dd14f7dbce766b0c022ab51 (127.29.124.134). New cstate: current_term: 1 leader_uuid: "9964c1c76dd14f7dbce766b0c022ab51" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } health_report { overall_health: UNKNOWN } } }
W20240501 14:04:01.919330 31790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:04:01.919651 31790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:04:01.927968 31790 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
I20240501 14:04:02.012706 31199 consensus_queue.cc:1035] T 51ed183d7ec2459faf1b25232b3e504d P 870fbdcfdac84bacae0ad22dba25f626 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:04:02.047646 31786 consensus_queue.cc:1035] T 51ed183d7ec2459faf1b25232b3e504d P 870fbdcfdac84bacae0ad22dba25f626 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:04:02.073402 31114 consensus_queue.cc:1035] T 1f5ff3b1c34a4de780fe4783b9b0eaf1 P a3e62ccad691413ea2554182f73418a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:04:02.098676 31114 consensus_queue.cc:1035] T 1f5ff3b1c34a4de780fe4783b9b0eaf1 P a3e62ccad691413ea2554182f73418a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:04:02.114094 31817 consensus_queue.cc:1035] T 4b44fc1c4c33499a90f6c446110d9c42 P 9964c1c76dd14f7dbce766b0c022ab51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:04:02.121574 30509 raft_consensus.cc:1261] T 4b44fc1c4c33499a90f6c446110d9c42 P 8883498098f4454bbfe563fb83d31899 [term 1 FOLLOWER]: Refusing update from remote peer 9964c1c76dd14f7dbce766b0c022ab51: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:04:02.122218 31817 consensus_queue.cc:1035] T 4b44fc1c4c33499a90f6c446110d9c42 P 9964c1c76dd14f7dbce766b0c022ab51 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 291.797 ms
time per row: 0.145899 ms
I20240501 14:04:02.368535 30228 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35960:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:04:02.378307 30357 tablet_service.cc:1467] Processing CreateTablet for tablet 66b6d18ec0cc4d8f8f2025666cff4205 (DEFAULT_TABLE table=rebalance_test_table_8 [id=55e37a75c21f4085860a913f735934de]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:02.378612 30620 tablet_service.cc:1467] Processing CreateTablet for tablet 66b6d18ec0cc4d8f8f2025666cff4205 (DEFAULT_TABLE table=rebalance_test_table_8 [id=55e37a75c21f4085860a913f735934de]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:02.378791 30357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66b6d18ec0cc4d8f8f2025666cff4205. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:02.379078 30620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66b6d18ec0cc4d8f8f2025666cff4205. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:02.379527 31012 tablet_service.cc:1467] Processing CreateTablet for tablet fc96027b13fe448894ccd93d6fa8a280 (DEFAULT_TABLE table=rebalance_test_table_8 [id=55e37a75c21f4085860a913f735934de]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:02.379900 31012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc96027b13fe448894ccd93d6fa8a280. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:02.381680 30882 tablet_service.cc:1467] Processing CreateTablet for tablet 66b6d18ec0cc4d8f8f2025666cff4205 (DEFAULT_TABLE table=rebalance_test_table_8 [id=55e37a75c21f4085860a913f735934de]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:02.382094 30882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66b6d18ec0cc4d8f8f2025666cff4205. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:02.382236 30488 tablet_service.cc:1467] Processing CreateTablet for tablet fc96027b13fe448894ccd93d6fa8a280 (DEFAULT_TABLE table=rebalance_test_table_8 [id=55e37a75c21f4085860a913f735934de]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:02.382652 30488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc96027b13fe448894ccd93d6fa8a280. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:02.382886 30750 tablet_service.cc:1467] Processing CreateTablet for tablet fc96027b13fe448894ccd93d6fa8a280 (DEFAULT_TABLE table=rebalance_test_table_8 [id=55e37a75c21f4085860a913f735934de]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:02.383337 30750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc96027b13fe448894ccd93d6fa8a280. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:02.386361 30358 tablet_service.cc:1467] Processing CreateTablet for tablet 63d9b49f760c401ab6b1d483e272729f (DEFAULT_TABLE table=rebalance_test_table_8 [id=55e37a75c21f4085860a913f735934de]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:02.386746 30358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63d9b49f760c401ab6b1d483e272729f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:02.389083 30881 tablet_service.cc:1467] Processing CreateTablet for tablet 63d9b49f760c401ab6b1d483e272729f (DEFAULT_TABLE table=rebalance_test_table_8 [id=55e37a75c21f4085860a913f735934de]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:02.389500 30881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63d9b49f760c401ab6b1d483e272729f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:02.394171 30619 tablet_service.cc:1467] Processing CreateTablet for tablet 63d9b49f760c401ab6b1d483e272729f (DEFAULT_TABLE table=rebalance_test_table_8 [id=55e37a75c21f4085860a913f735934de]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:02.394608 30619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63d9b49f760c401ab6b1d483e272729f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:02.399055 31864 tablet_bootstrap.cc:492] T fc96027b13fe448894ccd93d6fa8a280 P 870fbdcfdac84bacae0ad22dba25f626: Bootstrap starting.
I20240501 14:04:02.400816 31864 tablet_bootstrap.cc:654] T fc96027b13fe448894ccd93d6fa8a280 P 870fbdcfdac84bacae0ad22dba25f626: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:02.402747 31865 tablet_bootstrap.cc:492] T fc96027b13fe448894ccd93d6fa8a280 P 8883498098f4454bbfe563fb83d31899: Bootstrap starting.
I20240501 14:04:02.403920 31864 tablet_bootstrap.cc:492] T fc96027b13fe448894ccd93d6fa8a280 P 870fbdcfdac84bacae0ad22dba25f626: No bootstrap required, opened a new log
I20240501 14:04:02.404186 31864 ts_tablet_manager.cc:1367] T fc96027b13fe448894ccd93d6fa8a280 P 870fbdcfdac84bacae0ad22dba25f626: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 14:04:02.404978 31865 tablet_bootstrap.cc:654] T fc96027b13fe448894ccd93d6fa8a280 P 8883498098f4454bbfe563fb83d31899: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:02.404982 31864 raft_consensus.cc:348] T fc96027b13fe448894ccd93d6fa8a280 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:04:02.405248 31864 raft_consensus.cc:374] T fc96027b13fe448894ccd93d6fa8a280 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:02.405417 31864 raft_consensus.cc:729] T fc96027b13fe448894ccd93d6fa8a280 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 870fbdcfdac84bacae0ad22dba25f626, State: Initialized, Role: FOLLOWER
I20240501 14:04:02.405689 31864 consensus_queue.cc:260] T fc96027b13fe448894ccd93d6fa8a280 P 870fbdcfdac84bacae0ad22dba25f626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:04:02.406424 31864 ts_tablet_manager.cc:1398] T fc96027b13fe448894ccd93d6fa8a280 P 870fbdcfdac84bacae0ad22dba25f626: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:04:02.413815 31866 tablet_bootstrap.cc:492] T 66b6d18ec0cc4d8f8f2025666cff4205 P 99f6ca2d65d94b33b7a1d2341c24179a: Bootstrap starting.
I20240501 14:04:02.415511 31870 tablet_bootstrap.cc:492] T fc96027b13fe448894ccd93d6fa8a280 P 9964c1c76dd14f7dbce766b0c022ab51: Bootstrap starting.
I20240501 14:04:02.415632 31866 tablet_bootstrap.cc:654] T 66b6d18ec0cc4d8f8f2025666cff4205 P 99f6ca2d65d94b33b7a1d2341c24179a: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:02.417310 31870 tablet_bootstrap.cc:654] T fc96027b13fe448894ccd93d6fa8a280 P 9964c1c76dd14f7dbce766b0c022ab51: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:02.417779 31865 tablet_bootstrap.cc:492] T fc96027b13fe448894ccd93d6fa8a280 P 8883498098f4454bbfe563fb83d31899: No bootstrap required, opened a new log
I20240501 14:04:02.417842 31867 tablet_bootstrap.cc:492] T 66b6d18ec0cc4d8f8f2025666cff4205 P a3e62ccad691413ea2554182f73418a3: Bootstrap starting.
I20240501 14:04:02.419380 31865 ts_tablet_manager.cc:1367] T fc96027b13fe448894ccd93d6fa8a280 P 8883498098f4454bbfe563fb83d31899: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.004s
I20240501 14:04:02.420593 31865 raft_consensus.cc:348] T fc96027b13fe448894ccd93d6fa8a280 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:04:02.422072 31865 raft_consensus.cc:374] T fc96027b13fe448894ccd93d6fa8a280 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:02.422327 31865 raft_consensus.cc:729] T fc96027b13fe448894ccd93d6fa8a280 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8883498098f4454bbfe563fb83d31899, State: Initialized, Role: FOLLOWER
I20240501 14:04:02.422739 31867 tablet_bootstrap.cc:654] T 66b6d18ec0cc4d8f8f2025666cff4205 P a3e62ccad691413ea2554182f73418a3: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:02.422881 31865 consensus_queue.cc:260] T fc96027b13fe448894ccd93d6fa8a280 P 8883498098f4454bbfe563fb83d31899 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:04:02.423864 31865 ts_tablet_manager.cc:1398] T fc96027b13fe448894ccd93d6fa8a280 P 8883498098f4454bbfe563fb83d31899: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240501 14:04:02.425451 31866 tablet_bootstrap.cc:492] T 66b6d18ec0cc4d8f8f2025666cff4205 P 99f6ca2d65d94b33b7a1d2341c24179a: No bootstrap required, opened a new log
I20240501 14:04:02.425662 31866 ts_tablet_manager.cc:1367] T 66b6d18ec0cc4d8f8f2025666cff4205 P 99f6ca2d65d94b33b7a1d2341c24179a: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.009s
I20240501 14:04:02.426532 31870 tablet_bootstrap.cc:492] T fc96027b13fe448894ccd93d6fa8a280 P 9964c1c76dd14f7dbce766b0c022ab51: No bootstrap required, opened a new log
I20240501 14:04:02.426716 31870 ts_tablet_manager.cc:1367] T fc96027b13fe448894ccd93d6fa8a280 P 9964c1c76dd14f7dbce766b0c022ab51: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240501 14:04:02.427511 31870 raft_consensus.cc:348] T fc96027b13fe448894ccd93d6fa8a280 P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:04:02.427753 31870 raft_consensus.cc:374] T fc96027b13fe448894ccd93d6fa8a280 P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:02.427856 31870 raft_consensus.cc:729] T fc96027b13fe448894ccd93d6fa8a280 P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9964c1c76dd14f7dbce766b0c022ab51, State: Initialized, Role: FOLLOWER
I20240501 14:04:02.428140 31870 consensus_queue.cc:260] T fc96027b13fe448894ccd93d6fa8a280 P 9964c1c76dd14f7dbce766b0c022ab51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:04:02.428737 31870 ts_tablet_manager.cc:1398] T fc96027b13fe448894ccd93d6fa8a280 P 9964c1c76dd14f7dbce766b0c022ab51: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:04:02.430744 31866 raft_consensus.cc:348] T 66b6d18ec0cc4d8f8f2025666cff4205 P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:04:02.431020 31863 tablet_bootstrap.cc:492] T 66b6d18ec0cc4d8f8f2025666cff4205 P c0fad54e965647aaaa34b56674646cb4: Bootstrap starting.
I20240501 14:04:02.431099 31866 raft_consensus.cc:374] T 66b6d18ec0cc4d8f8f2025666cff4205 P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:02.431244 31866 raft_consensus.cc:729] T 66b6d18ec0cc4d8f8f2025666cff4205 P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99f6ca2d65d94b33b7a1d2341c24179a, State: Initialized, Role: FOLLOWER
I20240501 14:04:02.431519 31866 consensus_queue.cc:260] T 66b6d18ec0cc4d8f8f2025666cff4205 P 99f6ca2d65d94b33b7a1d2341c24179a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:04:02.432174 31866 ts_tablet_manager.cc:1398] T 66b6d18ec0cc4d8f8f2025666cff4205 P 99f6ca2d65d94b33b7a1d2341c24179a: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:04:02.432499 31866 tablet_bootstrap.cc:492] T 63d9b49f760c401ab6b1d483e272729f P 99f6ca2d65d94b33b7a1d2341c24179a: Bootstrap starting.
I20240501 14:04:02.432950 31863 tablet_bootstrap.cc:654] T 66b6d18ec0cc4d8f8f2025666cff4205 P c0fad54e965647aaaa34b56674646cb4: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:02.434202 31866 tablet_bootstrap.cc:654] T 63d9b49f760c401ab6b1d483e272729f P 99f6ca2d65d94b33b7a1d2341c24179a: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:02.435336 31867 tablet_bootstrap.cc:492] T 66b6d18ec0cc4d8f8f2025666cff4205 P a3e62ccad691413ea2554182f73418a3: No bootstrap required, opened a new log
I20240501 14:04:02.435552 31867 ts_tablet_manager.cc:1367] T 66b6d18ec0cc4d8f8f2025666cff4205 P a3e62ccad691413ea2554182f73418a3: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20240501 14:04:02.436100 31866 tablet_bootstrap.cc:492] T 63d9b49f760c401ab6b1d483e272729f P 99f6ca2d65d94b33b7a1d2341c24179a: No bootstrap required, opened a new log
I20240501 14:04:02.436313 31866 ts_tablet_manager.cc:1367] T 63d9b49f760c401ab6b1d483e272729f P 99f6ca2d65d94b33b7a1d2341c24179a: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240501 14:04:02.436470 31867 raft_consensus.cc:348] T 66b6d18ec0cc4d8f8f2025666cff4205 P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:04:02.436800 31867 raft_consensus.cc:374] T 66b6d18ec0cc4d8f8f2025666cff4205 P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:02.436944 31867 raft_consensus.cc:729] T 66b6d18ec0cc4d8f8f2025666cff4205 P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3e62ccad691413ea2554182f73418a3, State: Initialized, Role: FOLLOWER
I20240501 14:04:02.437084 31866 raft_consensus.cc:348] T 63d9b49f760c401ab6b1d483e272729f P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:04:02.437335 31866 raft_consensus.cc:374] T 63d9b49f760c401ab6b1d483e272729f P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:02.437223 31867 consensus_queue.cc:260] T 66b6d18ec0cc4d8f8f2025666cff4205 P a3e62ccad691413ea2554182f73418a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:04:02.437413 31866 raft_consensus.cc:729] T 63d9b49f760c401ab6b1d483e272729f P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99f6ca2d65d94b33b7a1d2341c24179a, State: Initialized, Role: FOLLOWER
I20240501 14:04:02.437773 31866 consensus_queue.cc:260] T 63d9b49f760c401ab6b1d483e272729f P 99f6ca2d65d94b33b7a1d2341c24179a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:04:02.438359 31867 ts_tablet_manager.cc:1398] T 66b6d18ec0cc4d8f8f2025666cff4205 P a3e62ccad691413ea2554182f73418a3: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:04:02.438647 31867 tablet_bootstrap.cc:492] T 63d9b49f760c401ab6b1d483e272729f P a3e62ccad691413ea2554182f73418a3: Bootstrap starting.
I20240501 14:04:02.441398 31867 tablet_bootstrap.cc:654] T 63d9b49f760c401ab6b1d483e272729f P a3e62ccad691413ea2554182f73418a3: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:02.441612 31863 tablet_bootstrap.cc:492] T 66b6d18ec0cc4d8f8f2025666cff4205 P c0fad54e965647aaaa34b56674646cb4: No bootstrap required, opened a new log
I20240501 14:04:02.441756 31863 ts_tablet_manager.cc:1367] T 66b6d18ec0cc4d8f8f2025666cff4205 P c0fad54e965647aaaa34b56674646cb4: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20240501 14:04:02.442157 31866 ts_tablet_manager.cc:1398] T 63d9b49f760c401ab6b1d483e272729f P 99f6ca2d65d94b33b7a1d2341c24179a: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240501 14:04:02.442790 31863 raft_consensus.cc:348] T 66b6d18ec0cc4d8f8f2025666cff4205 P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:04:02.443006 31863 raft_consensus.cc:374] T 66b6d18ec0cc4d8f8f2025666cff4205 P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:02.443099 31863 raft_consensus.cc:729] T 66b6d18ec0cc4d8f8f2025666cff4205 P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0fad54e965647aaaa34b56674646cb4, State: Initialized, Role: FOLLOWER
I20240501 14:04:02.443320 31863 consensus_queue.cc:260] T 66b6d18ec0cc4d8f8f2025666cff4205 P c0fad54e965647aaaa34b56674646cb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:04:02.443759 31863 ts_tablet_manager.cc:1398] T 66b6d18ec0cc4d8f8f2025666cff4205 P c0fad54e965647aaaa34b56674646cb4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:04:02.444365 31863 tablet_bootstrap.cc:492] T 63d9b49f760c401ab6b1d483e272729f P c0fad54e965647aaaa34b56674646cb4: Bootstrap starting.
I20240501 14:04:02.446357 31867 tablet_bootstrap.cc:492] T 63d9b49f760c401ab6b1d483e272729f P a3e62ccad691413ea2554182f73418a3: No bootstrap required, opened a new log
I20240501 14:04:02.446518 31867 ts_tablet_manager.cc:1367] T 63d9b49f760c401ab6b1d483e272729f P a3e62ccad691413ea2554182f73418a3: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.004s
I20240501 14:04:02.446604 31863 tablet_bootstrap.cc:654] T 63d9b49f760c401ab6b1d483e272729f P c0fad54e965647aaaa34b56674646cb4: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:02.447885 31867 raft_consensus.cc:348] T 63d9b49f760c401ab6b1d483e272729f P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:04:02.448174 31867 raft_consensus.cc:374] T 63d9b49f760c401ab6b1d483e272729f P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:02.448299 31867 raft_consensus.cc:729] T 63d9b49f760c401ab6b1d483e272729f P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3e62ccad691413ea2554182f73418a3, State: Initialized, Role: FOLLOWER
I20240501 14:04:02.448446 31863 tablet_bootstrap.cc:492] T 63d9b49f760c401ab6b1d483e272729f P c0fad54e965647aaaa34b56674646cb4: No bootstrap required, opened a new log
I20240501 14:04:02.448593 31863 ts_tablet_manager.cc:1367] T 63d9b49f760c401ab6b1d483e272729f P c0fad54e965647aaaa34b56674646cb4: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:04:02.448941 31867 consensus_queue.cc:260] T 63d9b49f760c401ab6b1d483e272729f P a3e62ccad691413ea2554182f73418a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:04:02.449137 31863 raft_consensus.cc:348] T 63d9b49f760c401ab6b1d483e272729f P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:04:02.449376 31863 raft_consensus.cc:374] T 63d9b49f760c401ab6b1d483e272729f P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:02.449462 31867 ts_tablet_manager.cc:1398] T 63d9b49f760c401ab6b1d483e272729f P a3e62ccad691413ea2554182f73418a3: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:04:02.449461 31863 raft_consensus.cc:729] T 63d9b49f760c401ab6b1d483e272729f P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0fad54e965647aaaa34b56674646cb4, State: Initialized, Role: FOLLOWER
I20240501 14:04:02.449685 31863 consensus_queue.cc:260] T 63d9b49f760c401ab6b1d483e272729f P c0fad54e965647aaaa34b56674646cb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:04:02.450203 31863 ts_tablet_manager.cc:1398] T 63d9b49f760c401ab6b1d483e272729f P c0fad54e965647aaaa34b56674646cb4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:04:02.499233 31199 raft_consensus.cc:482] T fc96027b13fe448894ccd93d6fa8a280 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:04:02.499433 31199 raft_consensus.cc:504] T fc96027b13fe448894ccd93d6fa8a280 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:04:02.500078 31199 leader_election.cc:288] T fc96027b13fe448894ccd93d6fa8a280 P 870fbdcfdac84bacae0ad22dba25f626 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8883498098f4454bbfe563fb83d31899 (127.29.124.130:43099), 9964c1c76dd14f7dbce766b0c022ab51 (127.29.124.134:38101)
I20240501 14:04:02.500567 30508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc96027b13fe448894ccd93d6fa8a280" candidate_uuid: "870fbdcfdac84bacae0ad22dba25f626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8883498098f4454bbfe563fb83d31899" is_pre_election: true
I20240501 14:04:02.500591 31032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc96027b13fe448894ccd93d6fa8a280" candidate_uuid: "870fbdcfdac84bacae0ad22dba25f626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9964c1c76dd14f7dbce766b0c022ab51" is_pre_election: true
I20240501 14:04:02.500782 31032 raft_consensus.cc:2454] T fc96027b13fe448894ccd93d6fa8a280 P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 870fbdcfdac84bacae0ad22dba25f626 in term 0.
I20240501 14:04:02.500792 30508 raft_consensus.cc:2454] T fc96027b13fe448894ccd93d6fa8a280 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 870fbdcfdac84bacae0ad22dba25f626 in term 0.
I20240501 14:04:02.501191 30704 leader_election.cc:302] T fc96027b13fe448894ccd93d6fa8a280 P 870fbdcfdac84bacae0ad22dba25f626 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 870fbdcfdac84bacae0ad22dba25f626, 9964c1c76dd14f7dbce766b0c022ab51; no voters:
I20240501 14:04:02.501502 31199 raft_consensus.cc:2781] T fc96027b13fe448894ccd93d6fa8a280 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:04:02.501600 31199 raft_consensus.cc:482] T fc96027b13fe448894ccd93d6fa8a280 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:04:02.501682 31199 raft_consensus.cc:3037] T fc96027b13fe448894ccd93d6fa8a280 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:02.503540 31199 raft_consensus.cc:504] T fc96027b13fe448894ccd93d6fa8a280 P 870fbdcfdac84bacae0ad22dba25f626 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:04:02.504066 31199 leader_election.cc:288] T fc96027b13fe448894ccd93d6fa8a280 P 870fbdcfdac84bacae0ad22dba25f626 [CANDIDATE]: Term 1 election: Requested vote from peers 8883498098f4454bbfe563fb83d31899 (127.29.124.130:43099), 9964c1c76dd14f7dbce766b0c022ab51 (127.29.124.134:38101)
I20240501 14:04:02.504578 30508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc96027b13fe448894ccd93d6fa8a280" candidate_uuid: "870fbdcfdac84bacae0ad22dba25f626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8883498098f4454bbfe563fb83d31899"
I20240501 14:04:02.504630 31032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc96027b13fe448894ccd93d6fa8a280" candidate_uuid: "870fbdcfdac84bacae0ad22dba25f626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9964c1c76dd14f7dbce766b0c022ab51"
I20240501 14:04:02.504771 30508 raft_consensus.cc:3037] T fc96027b13fe448894ccd93d6fa8a280 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:02.504796 31032 raft_consensus.cc:3037] T fc96027b13fe448894ccd93d6fa8a280 P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:02.506705 31032 raft_consensus.cc:2454] T fc96027b13fe448894ccd93d6fa8a280 P 9964c1c76dd14f7dbce766b0c022ab51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 870fbdcfdac84bacae0ad22dba25f626 in term 1.
I20240501 14:04:02.506772 30508 raft_consensus.cc:2454] T fc96027b13fe448894ccd93d6fa8a280 P 8883498098f4454bbfe563fb83d31899 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 870fbdcfdac84bacae0ad22dba25f626 in term 1.
I20240501 14:04:02.507155 30707 leader_election.cc:302] T fc96027b13fe448894ccd93d6fa8a280 P 870fbdcfdac84bacae0ad22dba25f626 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 870fbdcfdac84bacae0ad22dba25f626, 8883498098f4454bbfe563fb83d31899; no voters:
I20240501 14:04:02.507440 31199 raft_consensus.cc:2781] T fc96027b13fe448894ccd93d6fa8a280 P 870fbdcfdac84bacae0ad22dba25f626 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:04:02.507593 31199 raft_consensus.cc:686] T fc96027b13fe448894ccd93d6fa8a280 P 870fbdcfdac84bacae0ad22dba25f626 [term 1 LEADER]: Becoming Leader. State: Replica: 870fbdcfdac84bacae0ad22dba25f626, State: Running, Role: LEADER
I20240501 14:04:02.507855 31199 consensus_queue.cc:237] T fc96027b13fe448894ccd93d6fa8a280 P 870fbdcfdac84bacae0ad22dba25f626 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:04:02.511039 30228 catalog_manager.cc:5484] T fc96027b13fe448894ccd93d6fa8a280 P 870fbdcfdac84bacae0ad22dba25f626 reported cstate change: term changed from 0 to 1, leader changed from <none> to 870fbdcfdac84bacae0ad22dba25f626 (127.29.124.132). New cstate: current_term: 1 leader_uuid: "870fbdcfdac84bacae0ad22dba25f626" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } health_report { overall_health: HEALTHY } } }
I20240501 14:04:02.606864 31878 raft_consensus.cc:482] T 63d9b49f760c401ab6b1d483e272729f P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:04:02.607056 31878 raft_consensus.cc:504] T 63d9b49f760c401ab6b1d483e272729f P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:04:02.607754 31878 leader_election.cc:288] T 63d9b49f760c401ab6b1d483e272729f P 99f6ca2d65d94b33b7a1d2341c24179a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0fad54e965647aaaa34b56674646cb4 (127.29.124.129:42363), a3e62ccad691413ea2554182f73418a3 (127.29.124.131:43965)
I20240501 14:04:02.608273 30378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d9b49f760c401ab6b1d483e272729f" candidate_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0fad54e965647aaaa34b56674646cb4" is_pre_election: true
I20240501 14:04:02.608409 30639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d9b49f760c401ab6b1d483e272729f" candidate_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3e62ccad691413ea2554182f73418a3" is_pre_election: true
I20240501 14:04:02.608506 30378 raft_consensus.cc:2454] T 63d9b49f760c401ab6b1d483e272729f P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99f6ca2d65d94b33b7a1d2341c24179a in term 0.
I20240501 14:04:02.608647 30639 raft_consensus.cc:2454] T 63d9b49f760c401ab6b1d483e272729f P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99f6ca2d65d94b33b7a1d2341c24179a in term 0.
I20240501 14:04:02.608978 30836 leader_election.cc:302] T 63d9b49f760c401ab6b1d483e272729f P 99f6ca2d65d94b33b7a1d2341c24179a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99f6ca2d65d94b33b7a1d2341c24179a, c0fad54e965647aaaa34b56674646cb4; no voters:
I20240501 14:04:02.609294 31878 raft_consensus.cc:2781] T 63d9b49f760c401ab6b1d483e272729f P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:04:02.609408 31878 raft_consensus.cc:482] T 63d9b49f760c401ab6b1d483e272729f P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:04:02.609489 31878 raft_consensus.cc:3037] T 63d9b49f760c401ab6b1d483e272729f P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:02.611508 31878 raft_consensus.cc:504] T 63d9b49f760c401ab6b1d483e272729f P 99f6ca2d65d94b33b7a1d2341c24179a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:04:02.612094 31878 leader_election.cc:288] T 63d9b49f760c401ab6b1d483e272729f P 99f6ca2d65d94b33b7a1d2341c24179a [CANDIDATE]: Term 1 election: Requested vote from peers c0fad54e965647aaaa34b56674646cb4 (127.29.124.129:42363), a3e62ccad691413ea2554182f73418a3 (127.29.124.131:43965)
I20240501 14:04:02.612511 30639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d9b49f760c401ab6b1d483e272729f" candidate_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3e62ccad691413ea2554182f73418a3"
I20240501 14:04:02.612517 30378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63d9b49f760c401ab6b1d483e272729f" candidate_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0fad54e965647aaaa34b56674646cb4"
I20240501 14:04:02.612673 30378 raft_consensus.cc:3037] T 63d9b49f760c401ab6b1d483e272729f P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:02.612680 30639 raft_consensus.cc:3037] T 63d9b49f760c401ab6b1d483e272729f P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:02.614547 30639 raft_consensus.cc:2454] T 63d9b49f760c401ab6b1d483e272729f P a3e62ccad691413ea2554182f73418a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99f6ca2d65d94b33b7a1d2341c24179a in term 1.
I20240501 14:04:02.614552 30378 raft_consensus.cc:2454] T 63d9b49f760c401ab6b1d483e272729f P c0fad54e965647aaaa34b56674646cb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99f6ca2d65d94b33b7a1d2341c24179a in term 1.
I20240501 14:04:02.614982 30835 leader_election.cc:302] T 63d9b49f760c401ab6b1d483e272729f P 99f6ca2d65d94b33b7a1d2341c24179a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 99f6ca2d65d94b33b7a1d2341c24179a, a3e62ccad691413ea2554182f73418a3, c0fad54e965647aaaa34b56674646cb4; no voters:
I20240501 14:04:02.615254 31878 raft_consensus.cc:2781] T 63d9b49f760c401ab6b1d483e272729f P 99f6ca2d65d94b33b7a1d2341c24179a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:04:02.615388 31878 raft_consensus.cc:686] T 63d9b49f760c401ab6b1d483e272729f P 99f6ca2d65d94b33b7a1d2341c24179a [term 1 LEADER]: Becoming Leader. State: Replica: 99f6ca2d65d94b33b7a1d2341c24179a, State: Running, Role: LEADER
I20240501 14:04:02.615643 31878 consensus_queue.cc:237] T 63d9b49f760c401ab6b1d483e272729f P 99f6ca2d65d94b33b7a1d2341c24179a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:04:02.618467 30233 catalog_manager.cc:5484] T 63d9b49f760c401ab6b1d483e272729f P 99f6ca2d65d94b33b7a1d2341c24179a reported cstate change: term changed from 0 to 1, leader changed from <none> to 99f6ca2d65d94b33b7a1d2341c24179a (127.29.124.133). New cstate: current_term: 1 leader_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } health_report { overall_health: UNKNOWN } } }
I20240501 14:04:02.647778 31755 raft_consensus.cc:482] T 66b6d18ec0cc4d8f8f2025666cff4205 P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:04:02.647991 31755 raft_consensus.cc:504] T 66b6d18ec0cc4d8f8f2025666cff4205 P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:04:02.648770 31755 leader_election.cc:288] T 66b6d18ec0cc4d8f8f2025666cff4205 P c0fad54e965647aaaa34b56674646cb4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99f6ca2d65d94b33b7a1d2341c24179a (127.29.124.133:35195), a3e62ccad691413ea2554182f73418a3 (127.29.124.131:43965)
I20240501 14:04:02.649324 30639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66b6d18ec0cc4d8f8f2025666cff4205" candidate_uuid: "c0fad54e965647aaaa34b56674646cb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3e62ccad691413ea2554182f73418a3" is_pre_election: true
I20240501 14:04:02.649354 30901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66b6d18ec0cc4d8f8f2025666cff4205" candidate_uuid: "c0fad54e965647aaaa34b56674646cb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" is_pre_election: true
I20240501 14:04:02.649566 30901 raft_consensus.cc:2454] T 66b6d18ec0cc4d8f8f2025666cff4205 P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0fad54e965647aaaa34b56674646cb4 in term 0.
I20240501 14:04:02.649569 30639 raft_consensus.cc:2454] T 66b6d18ec0cc4d8f8f2025666cff4205 P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0fad54e965647aaaa34b56674646cb4 in term 0.
I20240501 14:04:02.650009 30311 leader_election.cc:302] T 66b6d18ec0cc4d8f8f2025666cff4205 P c0fad54e965647aaaa34b56674646cb4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 99f6ca2d65d94b33b7a1d2341c24179a, a3e62ccad691413ea2554182f73418a3, c0fad54e965647aaaa34b56674646cb4; no voters:
I20240501 14:04:02.650324 31755 raft_consensus.cc:2781] T 66b6d18ec0cc4d8f8f2025666cff4205 P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:04:02.650435 31755 raft_consensus.cc:482] T 66b6d18ec0cc4d8f8f2025666cff4205 P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:04:02.650511 31755 raft_consensus.cc:3037] T 66b6d18ec0cc4d8f8f2025666cff4205 P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:02.653350 31755 raft_consensus.cc:504] T 66b6d18ec0cc4d8f8f2025666cff4205 P c0fad54e965647aaaa34b56674646cb4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:04:02.654029 31755 leader_election.cc:288] T 66b6d18ec0cc4d8f8f2025666cff4205 P c0fad54e965647aaaa34b56674646cb4 [CANDIDATE]: Term 1 election: Requested vote from peers 99f6ca2d65d94b33b7a1d2341c24179a (127.29.124.133:35195), a3e62ccad691413ea2554182f73418a3 (127.29.124.131:43965)
I20240501 14:04:02.654513 30639 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66b6d18ec0cc4d8f8f2025666cff4205" candidate_uuid: "c0fad54e965647aaaa34b56674646cb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3e62ccad691413ea2554182f73418a3"
I20240501 14:04:02.654541 30901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66b6d18ec0cc4d8f8f2025666cff4205" candidate_uuid: "c0fad54e965647aaaa34b56674646cb4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99f6ca2d65d94b33b7a1d2341c24179a"
I20240501 14:04:02.654716 30901 raft_consensus.cc:3037] T 66b6d18ec0cc4d8f8f2025666cff4205 P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:02.654714 30639 raft_consensus.cc:3037] T 66b6d18ec0cc4d8f8f2025666cff4205 P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:02.656500 30639 raft_consensus.cc:2454] T 66b6d18ec0cc4d8f8f2025666cff4205 P a3e62ccad691413ea2554182f73418a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0fad54e965647aaaa34b56674646cb4 in term 1.
I20240501 14:04:02.656910 30311 leader_election.cc:302] T 66b6d18ec0cc4d8f8f2025666cff4205 P c0fad54e965647aaaa34b56674646cb4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3e62ccad691413ea2554182f73418a3, c0fad54e965647aaaa34b56674646cb4; no voters:
I20240501 14:04:02.657228 31755 raft_consensus.cc:2781] T 66b6d18ec0cc4d8f8f2025666cff4205 P c0fad54e965647aaaa34b56674646cb4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:04:02.657644 31755 raft_consensus.cc:686] T 66b6d18ec0cc4d8f8f2025666cff4205 P c0fad54e965647aaaa34b56674646cb4 [term 1 LEADER]: Becoming Leader. State: Replica: c0fad54e965647aaaa34b56674646cb4, State: Running, Role: LEADER
I20240501 14:04:02.657979 31755 consensus_queue.cc:237] T 66b6d18ec0cc4d8f8f2025666cff4205 P c0fad54e965647aaaa34b56674646cb4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:04:02.665585 30233 catalog_manager.cc:5484] T 66b6d18ec0cc4d8f8f2025666cff4205 P c0fad54e965647aaaa34b56674646cb4 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0fad54e965647aaaa34b56674646cb4 (127.29.124.129). New cstate: current_term: 1 leader_uuid: "c0fad54e965647aaaa34b56674646cb4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } health_report { overall_health: UNKNOWN } } }
I20240501 14:04:02.687098 30901 raft_consensus.cc:2454] T 66b6d18ec0cc4d8f8f2025666cff4205 P 99f6ca2d65d94b33b7a1d2341c24179a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0fad54e965647aaaa34b56674646cb4 in term 1.
I20240501 14:04:02.920650 31199 consensus_queue.cc:1035] T fc96027b13fe448894ccd93d6fa8a280 P 870fbdcfdac84bacae0ad22dba25f626 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:04:02.932520 31199 consensus_queue.cc:1035] T fc96027b13fe448894ccd93d6fa8a280 P 870fbdcfdac84bacae0ad22dba25f626 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:04:03.034175 31878 consensus_queue.cc:1035] T 63d9b49f760c401ab6b1d483e272729f P 99f6ca2d65d94b33b7a1d2341c24179a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:04:03.037400 31883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:04:03.037858 31883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 14:04:03.040707 31878 consensus_queue.cc:1035] T 63d9b49f760c401ab6b1d483e272729f P 99f6ca2d65d94b33b7a1d2341c24179a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:04:03.049862 31883 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:04:03.055101 31755 consensus_queue.cc:1035] T 66b6d18ec0cc4d8f8f2025666cff4205 P c0fad54e965647aaaa34b56674646cb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:04:03.059350 31755 consensus_queue.cc:1035] T 66b6d18ec0cc4d8f8f2025666cff4205 P c0fad54e965647aaaa34b56674646cb4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_8'
INSERT report
rows total: 2000
time total: 224.503 ms
time per row: 0.112252 ms
I20240501 14:04:03.457067 30233 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35960:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:04:03.465852 31012 tablet_service.cc:1467] Processing CreateTablet for tablet eada160f92e446468cb1a1c78ef4790a (DEFAULT_TABLE table=rebalance_test_table_9 [id=437997318ca84765afd1fbea86896301]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:03.466341 31012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eada160f92e446468cb1a1c78ef4790a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:03.466498 30750 tablet_service.cc:1467] Processing CreateTablet for tablet eada160f92e446468cb1a1c78ef4790a (DEFAULT_TABLE table=rebalance_test_table_9 [id=437997318ca84765afd1fbea86896301]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:03.466881 30750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eada160f92e446468cb1a1c78ef4790a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:03.467566 30620 tablet_service.cc:1467] Processing CreateTablet for tablet 850c9d8997af4db8b1d28707dbd517cb (DEFAULT_TABLE table=rebalance_test_table_9 [id=437997318ca84765afd1fbea86896301]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:03.468005 30620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 850c9d8997af4db8b1d28707dbd517cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:03.469658 30751 tablet_service.cc:1467] Processing CreateTablet for tablet 8544c28e5c1448e8a069c1842d6638d9 (DEFAULT_TABLE table=rebalance_test_table_9 [id=437997318ca84765afd1fbea86896301]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:03.470052 30751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8544c28e5c1448e8a069c1842d6638d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:03.470258 30357 tablet_service.cc:1467] Processing CreateTablet for tablet 850c9d8997af4db8b1d28707dbd517cb (DEFAULT_TABLE table=rebalance_test_table_9 [id=437997318ca84765afd1fbea86896301]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:03.470674 30357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 850c9d8997af4db8b1d28707dbd517cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:03.474314 30488 tablet_service.cc:1467] Processing CreateTablet for tablet eada160f92e446468cb1a1c78ef4790a (DEFAULT_TABLE table=rebalance_test_table_9 [id=437997318ca84765afd1fbea86896301]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:03.475054 30489 tablet_service.cc:1467] Processing CreateTablet for tablet 8544c28e5c1448e8a069c1842d6638d9 (DEFAULT_TABLE table=rebalance_test_table_9 [id=437997318ca84765afd1fbea86896301]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:03.475430 30489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8544c28e5c1448e8a069c1842d6638d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:03.477584 31013 tablet_service.cc:1467] Processing CreateTablet for tablet 8544c28e5c1448e8a069c1842d6638d9 (DEFAULT_TABLE table=rebalance_test_table_9 [id=437997318ca84765afd1fbea86896301]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:03.478008 31013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8544c28e5c1448e8a069c1842d6638d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:03.481853 31951 tablet_bootstrap.cc:492] T eada160f92e446468cb1a1c78ef4790a P 870fbdcfdac84bacae0ad22dba25f626: Bootstrap starting.
I20240501 14:04:03.482311 30488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eada160f92e446468cb1a1c78ef4790a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:03.482550 30882 tablet_service.cc:1467] Processing CreateTablet for tablet 850c9d8997af4db8b1d28707dbd517cb (DEFAULT_TABLE table=rebalance_test_table_9 [id=437997318ca84765afd1fbea86896301]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:03.483027 30882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 850c9d8997af4db8b1d28707dbd517cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:03.483625 31952 tablet_bootstrap.cc:492] T 850c9d8997af4db8b1d28707dbd517cb P a3e62ccad691413ea2554182f73418a3: Bootstrap starting.
I20240501 14:04:03.483678 31951 tablet_bootstrap.cc:654] T eada160f92e446468cb1a1c78ef4790a P 870fbdcfdac84bacae0ad22dba25f626: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:03.486933 31952 tablet_bootstrap.cc:654] T 850c9d8997af4db8b1d28707dbd517cb P a3e62ccad691413ea2554182f73418a3: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:03.504002 31957 tablet_bootstrap.cc:492] T 850c9d8997af4db8b1d28707dbd517cb P c0fad54e965647aaaa34b56674646cb4: Bootstrap starting.
I20240501 14:04:03.510941 31957 tablet_bootstrap.cc:654] T 850c9d8997af4db8b1d28707dbd517cb P c0fad54e965647aaaa34b56674646cb4: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:03.511188 31952 tablet_bootstrap.cc:492] T 850c9d8997af4db8b1d28707dbd517cb P a3e62ccad691413ea2554182f73418a3: No bootstrap required, opened a new log
I20240501 14:04:03.511426 31952 ts_tablet_manager.cc:1367] T 850c9d8997af4db8b1d28707dbd517cb P a3e62ccad691413ea2554182f73418a3: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.023s
I20240501 14:04:03.512354 31952 raft_consensus.cc:348] T 850c9d8997af4db8b1d28707dbd517cb P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:04:03.512776 31952 raft_consensus.cc:374] T 850c9d8997af4db8b1d28707dbd517cb P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:03.512919 31952 raft_consensus.cc:729] T 850c9d8997af4db8b1d28707dbd517cb P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3e62ccad691413ea2554182f73418a3, State: Initialized, Role: FOLLOWER
I20240501 14:04:03.513288 31952 consensus_queue.cc:260] T 850c9d8997af4db8b1d28707dbd517cb P a3e62ccad691413ea2554182f73418a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:04:03.514874 31952 ts_tablet_manager.cc:1398] T 850c9d8997af4db8b1d28707dbd517cb P a3e62ccad691413ea2554182f73418a3: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:04:03.519436 31956 tablet_bootstrap.cc:492] T eada160f92e446468cb1a1c78ef4790a P 9964c1c76dd14f7dbce766b0c022ab51: Bootstrap starting.
I20240501 14:04:03.521360 31956 tablet_bootstrap.cc:654] T eada160f92e446468cb1a1c78ef4790a P 9964c1c76dd14f7dbce766b0c022ab51: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:03.521978 31958 tablet_bootstrap.cc:492] T 8544c28e5c1448e8a069c1842d6638d9 P 8883498098f4454bbfe563fb83d31899: Bootstrap starting.
I20240501 14:04:03.522270 31957 tablet_bootstrap.cc:492] T 850c9d8997af4db8b1d28707dbd517cb P c0fad54e965647aaaa34b56674646cb4: No bootstrap required, opened a new log
I20240501 14:04:03.522751 31957 ts_tablet_manager.cc:1367] T 850c9d8997af4db8b1d28707dbd517cb P c0fad54e965647aaaa34b56674646cb4: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.008s
I20240501 14:04:03.523670 31957 raft_consensus.cc:348] T 850c9d8997af4db8b1d28707dbd517cb P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:04:03.523928 31957 raft_consensus.cc:374] T 850c9d8997af4db8b1d28707dbd517cb P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:03.524031 31957 raft_consensus.cc:729] T 850c9d8997af4db8b1d28707dbd517cb P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0fad54e965647aaaa34b56674646cb4, State: Initialized, Role: FOLLOWER
I20240501 14:04:03.524341 31957 consensus_queue.cc:260] T 850c9d8997af4db8b1d28707dbd517cb P c0fad54e965647aaaa34b56674646cb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:04:03.524699 31958 tablet_bootstrap.cc:654] T 8544c28e5c1448e8a069c1842d6638d9 P 8883498098f4454bbfe563fb83d31899: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:03.525851 31957 ts_tablet_manager.cc:1398] T 850c9d8997af4db8b1d28707dbd517cb P c0fad54e965647aaaa34b56674646cb4: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:04:03.531088 31955 tablet_bootstrap.cc:492] T 850c9d8997af4db8b1d28707dbd517cb P 99f6ca2d65d94b33b7a1d2341c24179a: Bootstrap starting.
I20240501 14:04:03.533010 31955 tablet_bootstrap.cc:654] T 850c9d8997af4db8b1d28707dbd517cb P 99f6ca2d65d94b33b7a1d2341c24179a: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:03.533812 31956 tablet_bootstrap.cc:492] T eada160f92e446468cb1a1c78ef4790a P 9964c1c76dd14f7dbce766b0c022ab51: No bootstrap required, opened a new log
I20240501 14:04:03.533977 31956 ts_tablet_manager.cc:1367] T eada160f92e446468cb1a1c78ef4790a P 9964c1c76dd14f7dbce766b0c022ab51: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.003s
I20240501 14:04:03.534791 31956 raft_consensus.cc:348] T eada160f92e446468cb1a1c78ef4790a P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:04:03.534965 31956 raft_consensus.cc:374] T eada160f92e446468cb1a1c78ef4790a P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:03.535064 31956 raft_consensus.cc:729] T eada160f92e446468cb1a1c78ef4790a P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9964c1c76dd14f7dbce766b0c022ab51, State: Initialized, Role: FOLLOWER
I20240501 14:04:03.535148 31955 tablet_bootstrap.cc:492] T 850c9d8997af4db8b1d28707dbd517cb P 99f6ca2d65d94b33b7a1d2341c24179a: No bootstrap required, opened a new log
I20240501 14:04:03.535357 31955 ts_tablet_manager.cc:1367] T 850c9d8997af4db8b1d28707dbd517cb P 99f6ca2d65d94b33b7a1d2341c24179a: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.004s
I20240501 14:04:03.535276 31956 consensus_queue.cc:260] T eada160f92e446468cb1a1c78ef4790a P 9964c1c76dd14f7dbce766b0c022ab51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:04:03.535892 31956 ts_tablet_manager.cc:1398] T eada160f92e446468cb1a1c78ef4790a P 9964c1c76dd14f7dbce766b0c022ab51: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:04:03.536209 31956 tablet_bootstrap.cc:492] T 8544c28e5c1448e8a069c1842d6638d9 P 9964c1c76dd14f7dbce766b0c022ab51: Bootstrap starting.
I20240501 14:04:03.536144 31955 raft_consensus.cc:348] T 850c9d8997af4db8b1d28707dbd517cb P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:04:03.536375 31955 raft_consensus.cc:374] T 850c9d8997af4db8b1d28707dbd517cb P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:03.536473 31955 raft_consensus.cc:729] T 850c9d8997af4db8b1d28707dbd517cb P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99f6ca2d65d94b33b7a1d2341c24179a, State: Initialized, Role: FOLLOWER
I20240501 14:04:03.536701 31955 consensus_queue.cc:260] T 850c9d8997af4db8b1d28707dbd517cb P 99f6ca2d65d94b33b7a1d2341c24179a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:04:03.537457 31955 ts_tablet_manager.cc:1398] T 850c9d8997af4db8b1d28707dbd517cb P 99f6ca2d65d94b33b7a1d2341c24179a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:04:03.538715 31958 tablet_bootstrap.cc:492] T 8544c28e5c1448e8a069c1842d6638d9 P 8883498098f4454bbfe563fb83d31899: No bootstrap required, opened a new log
I20240501 14:04:03.539023 31951 tablet_bootstrap.cc:492] T eada160f92e446468cb1a1c78ef4790a P 870fbdcfdac84bacae0ad22dba25f626: No bootstrap required, opened a new log
I20240501 14:04:03.539222 31958 ts_tablet_manager.cc:1367] T 8544c28e5c1448e8a069c1842d6638d9 P 8883498098f4454bbfe563fb83d31899: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20240501 14:04:03.539228 31951 ts_tablet_manager.cc:1367] T eada160f92e446468cb1a1c78ef4790a P 870fbdcfdac84bacae0ad22dba25f626: Time spent bootstrapping tablet: real 0.057s user 0.023s sys 0.027s
I20240501 14:04:03.540064 31951 raft_consensus.cc:348] T eada160f92e446468cb1a1c78ef4790a P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:04:03.540307 31951 raft_consensus.cc:374] T eada160f92e446468cb1a1c78ef4790a P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:03.540082 31958 raft_consensus.cc:348] T 8544c28e5c1448e8a069c1842d6638d9 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:04:03.540438 31951 raft_consensus.cc:729] T eada160f92e446468cb1a1c78ef4790a P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 870fbdcfdac84bacae0ad22dba25f626, State: Initialized, Role: FOLLOWER
I20240501 14:04:03.540491 31958 raft_consensus.cc:374] T 8544c28e5c1448e8a069c1842d6638d9 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:03.540691 31958 raft_consensus.cc:729] T 8544c28e5c1448e8a069c1842d6638d9 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8883498098f4454bbfe563fb83d31899, State: Initialized, Role: FOLLOWER
I20240501 14:04:03.540777 31956 tablet_bootstrap.cc:654] T 8544c28e5c1448e8a069c1842d6638d9 P 9964c1c76dd14f7dbce766b0c022ab51: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:03.540746 31951 consensus_queue.cc:260] T eada160f92e446468cb1a1c78ef4790a P 870fbdcfdac84bacae0ad22dba25f626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:04:03.541000 31958 consensus_queue.cc:260] T 8544c28e5c1448e8a069c1842d6638d9 P 8883498098f4454bbfe563fb83d31899 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:04:03.542446 31951 ts_tablet_manager.cc:1398] T eada160f92e446468cb1a1c78ef4790a P 870fbdcfdac84bacae0ad22dba25f626: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:04:03.542605 31958 ts_tablet_manager.cc:1398] T 8544c28e5c1448e8a069c1842d6638d9 P 8883498098f4454bbfe563fb83d31899: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:04:03.542748 31951 tablet_bootstrap.cc:492] T 8544c28e5c1448e8a069c1842d6638d9 P 870fbdcfdac84bacae0ad22dba25f626: Bootstrap starting.
I20240501 14:04:03.542990 31958 tablet_bootstrap.cc:492] T eada160f92e446468cb1a1c78ef4790a P 8883498098f4454bbfe563fb83d31899: Bootstrap starting.
I20240501 14:04:03.545373 31958 tablet_bootstrap.cc:654] T eada160f92e446468cb1a1c78ef4790a P 8883498098f4454bbfe563fb83d31899: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:03.545683 31951 tablet_bootstrap.cc:654] T 8544c28e5c1448e8a069c1842d6638d9 P 870fbdcfdac84bacae0ad22dba25f626: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:03.550853 31956 tablet_bootstrap.cc:492] T 8544c28e5c1448e8a069c1842d6638d9 P 9964c1c76dd14f7dbce766b0c022ab51: No bootstrap required, opened a new log
I20240501 14:04:03.551062 31956 ts_tablet_manager.cc:1367] T 8544c28e5c1448e8a069c1842d6638d9 P 9964c1c76dd14f7dbce766b0c022ab51: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.006s
I20240501 14:04:03.551810 31951 tablet_bootstrap.cc:492] T 8544c28e5c1448e8a069c1842d6638d9 P 870fbdcfdac84bacae0ad22dba25f626: No bootstrap required, opened a new log
I20240501 14:04:03.551995 31951 ts_tablet_manager.cc:1367] T 8544c28e5c1448e8a069c1842d6638d9 P 870fbdcfdac84bacae0ad22dba25f626: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20240501 14:04:03.551910 31956 raft_consensus.cc:348] T 8544c28e5c1448e8a069c1842d6638d9 P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:04:03.552182 31956 raft_consensus.cc:374] T 8544c28e5c1448e8a069c1842d6638d9 P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:03.552274 31956 raft_consensus.cc:729] T 8544c28e5c1448e8a069c1842d6638d9 P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9964c1c76dd14f7dbce766b0c022ab51, State: Initialized, Role: FOLLOWER
I20240501 14:04:03.552524 31956 consensus_queue.cc:260] T 8544c28e5c1448e8a069c1842d6638d9 P 9964c1c76dd14f7dbce766b0c022ab51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:04:03.552740 31951 raft_consensus.cc:348] T 8544c28e5c1448e8a069c1842d6638d9 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:04:03.552947 31951 raft_consensus.cc:374] T 8544c28e5c1448e8a069c1842d6638d9 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:03.553025 31951 raft_consensus.cc:729] T 8544c28e5c1448e8a069c1842d6638d9 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 870fbdcfdac84bacae0ad22dba25f626, State: Initialized, Role: FOLLOWER
I20240501 14:04:03.553103 31956 ts_tablet_manager.cc:1398] T 8544c28e5c1448e8a069c1842d6638d9 P 9964c1c76dd14f7dbce766b0c022ab51: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:04:03.553309 31951 consensus_queue.cc:260] T 8544c28e5c1448e8a069c1842d6638d9 P 870fbdcfdac84bacae0ad22dba25f626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:04:03.553647 31958 tablet_bootstrap.cc:492] T eada160f92e446468cb1a1c78ef4790a P 8883498098f4454bbfe563fb83d31899: No bootstrap required, opened a new log
I20240501 14:04:03.553790 31951 ts_tablet_manager.cc:1398] T 8544c28e5c1448e8a069c1842d6638d9 P 870fbdcfdac84bacae0ad22dba25f626: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:04:03.553831 31958 ts_tablet_manager.cc:1367] T eada160f92e446468cb1a1c78ef4790a P 8883498098f4454bbfe563fb83d31899: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20240501 14:04:03.554554 31958 raft_consensus.cc:348] T eada160f92e446468cb1a1c78ef4790a P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:04:03.554771 31958 raft_consensus.cc:374] T eada160f92e446468cb1a1c78ef4790a P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:03.554912 31958 raft_consensus.cc:729] T eada160f92e446468cb1a1c78ef4790a P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8883498098f4454bbfe563fb83d31899, State: Initialized, Role: FOLLOWER
I20240501 14:04:03.555256 31958 consensus_queue.cc:260] T eada160f92e446468cb1a1c78ef4790a P 8883498098f4454bbfe563fb83d31899 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:04:03.556589 31958 ts_tablet_manager.cc:1398] T eada160f92e446468cb1a1c78ef4790a P 8883498098f4454bbfe563fb83d31899: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:04:03.626287 31199 raft_consensus.cc:482] T eada160f92e446468cb1a1c78ef4790a P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:04:03.626482 31199 raft_consensus.cc:504] T eada160f92e446468cb1a1c78ef4790a P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:04:03.627108 31199 leader_election.cc:288] T eada160f92e446468cb1a1c78ef4790a P 870fbdcfdac84bacae0ad22dba25f626 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9964c1c76dd14f7dbce766b0c022ab51 (127.29.124.134:38101), 8883498098f4454bbfe563fb83d31899 (127.29.124.130:43099)
I20240501 14:04:03.627586 31032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eada160f92e446468cb1a1c78ef4790a" candidate_uuid: "870fbdcfdac84bacae0ad22dba25f626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9964c1c76dd14f7dbce766b0c022ab51" is_pre_election: true
I20240501 14:04:03.627606 30508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eada160f92e446468cb1a1c78ef4790a" candidate_uuid: "870fbdcfdac84bacae0ad22dba25f626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8883498098f4454bbfe563fb83d31899" is_pre_election: true
I20240501 14:04:03.627804 30508 raft_consensus.cc:2454] T eada160f92e446468cb1a1c78ef4790a P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 870fbdcfdac84bacae0ad22dba25f626 in term 0.
I20240501 14:04:03.627813 31032 raft_consensus.cc:2454] T eada160f92e446468cb1a1c78ef4790a P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 870fbdcfdac84bacae0ad22dba25f626 in term 0.
I20240501 14:04:03.628254 30707 leader_election.cc:302] T eada160f92e446468cb1a1c78ef4790a P 870fbdcfdac84bacae0ad22dba25f626 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 870fbdcfdac84bacae0ad22dba25f626, 8883498098f4454bbfe563fb83d31899; no voters:
I20240501 14:04:03.628576 31199 raft_consensus.cc:2781] T eada160f92e446468cb1a1c78ef4790a P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:04:03.628683 31199 raft_consensus.cc:482] T eada160f92e446468cb1a1c78ef4790a P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:04:03.628798 31199 raft_consensus.cc:3037] T eada160f92e446468cb1a1c78ef4790a P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:03.631119 31199 raft_consensus.cc:504] T eada160f92e446468cb1a1c78ef4790a P 870fbdcfdac84bacae0ad22dba25f626 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:04:03.631655 31199 leader_election.cc:288] T eada160f92e446468cb1a1c78ef4790a P 870fbdcfdac84bacae0ad22dba25f626 [CANDIDATE]: Term 1 election: Requested vote from peers 9964c1c76dd14f7dbce766b0c022ab51 (127.29.124.134:38101), 8883498098f4454bbfe563fb83d31899 (127.29.124.130:43099)
I20240501 14:04:03.632099 31032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eada160f92e446468cb1a1c78ef4790a" candidate_uuid: "870fbdcfdac84bacae0ad22dba25f626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9964c1c76dd14f7dbce766b0c022ab51"
I20240501 14:04:03.632298 31032 raft_consensus.cc:3037] T eada160f92e446468cb1a1c78ef4790a P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:03.632333 30508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eada160f92e446468cb1a1c78ef4790a" candidate_uuid: "870fbdcfdac84bacae0ad22dba25f626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8883498098f4454bbfe563fb83d31899"
I20240501 14:04:03.632524 30508 raft_consensus.cc:3037] T eada160f92e446468cb1a1c78ef4790a P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:03.634552 31032 raft_consensus.cc:2454] T eada160f92e446468cb1a1c78ef4790a P 9964c1c76dd14f7dbce766b0c022ab51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 870fbdcfdac84bacae0ad22dba25f626 in term 1.
I20240501 14:04:03.634708 30508 raft_consensus.cc:2454] T eada160f92e446468cb1a1c78ef4790a P 8883498098f4454bbfe563fb83d31899 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 870fbdcfdac84bacae0ad22dba25f626 in term 1.
I20240501 14:04:03.635076 30704 leader_election.cc:302] T eada160f92e446468cb1a1c78ef4790a P 870fbdcfdac84bacae0ad22dba25f626 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 870fbdcfdac84bacae0ad22dba25f626, 9964c1c76dd14f7dbce766b0c022ab51; no voters:
I20240501 14:04:03.635367 31199 raft_consensus.cc:2781] T eada160f92e446468cb1a1c78ef4790a P 870fbdcfdac84bacae0ad22dba25f626 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:04:03.635522 31199 raft_consensus.cc:686] T eada160f92e446468cb1a1c78ef4790a P 870fbdcfdac84bacae0ad22dba25f626 [term 1 LEADER]: Becoming Leader. State: Replica: 870fbdcfdac84bacae0ad22dba25f626, State: Running, Role: LEADER
I20240501 14:04:03.635794 31199 consensus_queue.cc:237] T eada160f92e446468cb1a1c78ef4790a P 870fbdcfdac84bacae0ad22dba25f626 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:04:03.639042 30228 catalog_manager.cc:5484] T eada160f92e446468cb1a1c78ef4790a P 870fbdcfdac84bacae0ad22dba25f626 reported cstate change: term changed from 0 to 1, leader changed from <none> to 870fbdcfdac84bacae0ad22dba25f626 (127.29.124.132). New cstate: current_term: 1 leader_uuid: "870fbdcfdac84bacae0ad22dba25f626" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } health_report { overall_health: HEALTHY } } }
I20240501 14:04:03.691143 31114 raft_consensus.cc:482] T 850c9d8997af4db8b1d28707dbd517cb P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:04:03.691346 31114 raft_consensus.cc:504] T 850c9d8997af4db8b1d28707dbd517cb P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:04:03.692083 31114 leader_election.cc:288] T 850c9d8997af4db8b1d28707dbd517cb P a3e62ccad691413ea2554182f73418a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0fad54e965647aaaa34b56674646cb4 (127.29.124.129:42363), 99f6ca2d65d94b33b7a1d2341c24179a (127.29.124.133:35195)
I20240501 14:04:03.692548 30377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "850c9d8997af4db8b1d28707dbd517cb" candidate_uuid: "a3e62ccad691413ea2554182f73418a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0fad54e965647aaaa34b56674646cb4" is_pre_election: true
I20240501 14:04:03.692679 30901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "850c9d8997af4db8b1d28707dbd517cb" candidate_uuid: "a3e62ccad691413ea2554182f73418a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" is_pre_election: true
I20240501 14:04:03.692763 30377 raft_consensus.cc:2454] T 850c9d8997af4db8b1d28707dbd517cb P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3e62ccad691413ea2554182f73418a3 in term 0.
I20240501 14:04:03.692901 30901 raft_consensus.cc:2454] T 850c9d8997af4db8b1d28707dbd517cb P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3e62ccad691413ea2554182f73418a3 in term 0.
I20240501 14:04:03.693291 30574 leader_election.cc:302] T 850c9d8997af4db8b1d28707dbd517cb P a3e62ccad691413ea2554182f73418a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3e62ccad691413ea2554182f73418a3, c0fad54e965647aaaa34b56674646cb4; no voters:
I20240501 14:04:03.693557 31114 raft_consensus.cc:2781] T 850c9d8997af4db8b1d28707dbd517cb P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:04:03.693673 31114 raft_consensus.cc:482] T 850c9d8997af4db8b1d28707dbd517cb P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:04:03.693751 31114 raft_consensus.cc:3037] T 850c9d8997af4db8b1d28707dbd517cb P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:03.695557 31114 raft_consensus.cc:504] T 850c9d8997af4db8b1d28707dbd517cb P a3e62ccad691413ea2554182f73418a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:04:03.696141 31114 leader_election.cc:288] T 850c9d8997af4db8b1d28707dbd517cb P a3e62ccad691413ea2554182f73418a3 [CANDIDATE]: Term 1 election: Requested vote from peers c0fad54e965647aaaa34b56674646cb4 (127.29.124.129:42363), 99f6ca2d65d94b33b7a1d2341c24179a (127.29.124.133:35195)
I20240501 14:04:03.696485 30377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "850c9d8997af4db8b1d28707dbd517cb" candidate_uuid: "a3e62ccad691413ea2554182f73418a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0fad54e965647aaaa34b56674646cb4"
I20240501 14:04:03.696571 30901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "850c9d8997af4db8b1d28707dbd517cb" candidate_uuid: "a3e62ccad691413ea2554182f73418a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99f6ca2d65d94b33b7a1d2341c24179a"
I20240501 14:04:03.696681 30377 raft_consensus.cc:3037] T 850c9d8997af4db8b1d28707dbd517cb P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:03.696771 30901 raft_consensus.cc:3037] T 850c9d8997af4db8b1d28707dbd517cb P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:03.698829 30377 raft_consensus.cc:2454] T 850c9d8997af4db8b1d28707dbd517cb P c0fad54e965647aaaa34b56674646cb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3e62ccad691413ea2554182f73418a3 in term 1.
I20240501 14:04:03.698958 30901 raft_consensus.cc:2454] T 850c9d8997af4db8b1d28707dbd517cb P 99f6ca2d65d94b33b7a1d2341c24179a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3e62ccad691413ea2554182f73418a3 in term 1.
I20240501 14:04:03.699224 30574 leader_election.cc:302] T 850c9d8997af4db8b1d28707dbd517cb P a3e62ccad691413ea2554182f73418a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a3e62ccad691413ea2554182f73418a3, c0fad54e965647aaaa34b56674646cb4; no voters:
I20240501 14:04:03.699537 31114 raft_consensus.cc:2781] T 850c9d8997af4db8b1d28707dbd517cb P a3e62ccad691413ea2554182f73418a3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:04:03.699903 31114 raft_consensus.cc:686] T 850c9d8997af4db8b1d28707dbd517cb P a3e62ccad691413ea2554182f73418a3 [term 1 LEADER]: Becoming Leader. State: Replica: a3e62ccad691413ea2554182f73418a3, State: Running, Role: LEADER
I20240501 14:04:03.700181 31114 consensus_queue.cc:237] T 850c9d8997af4db8b1d28707dbd517cb P a3e62ccad691413ea2554182f73418a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:04:03.703210 30228 catalog_manager.cc:5484] T 850c9d8997af4db8b1d28707dbd517cb P a3e62ccad691413ea2554182f73418a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to a3e62ccad691413ea2554182f73418a3 (127.29.124.131). New cstate: current_term: 1 leader_uuid: "a3e62ccad691413ea2554182f73418a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } health_report { overall_health: HEALTHY } } }
I20240501 14:04:03.732416 31199 raft_consensus.cc:482] T 8544c28e5c1448e8a069c1842d6638d9 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:04:03.732610 31199 raft_consensus.cc:504] T 8544c28e5c1448e8a069c1842d6638d9 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:04:03.733317 31199 leader_election.cc:288] T 8544c28e5c1448e8a069c1842d6638d9 P 870fbdcfdac84bacae0ad22dba25f626 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9964c1c76dd14f7dbce766b0c022ab51 (127.29.124.134:38101), 8883498098f4454bbfe563fb83d31899 (127.29.124.130:43099)
I20240501 14:04:03.733847 31032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8544c28e5c1448e8a069c1842d6638d9" candidate_uuid: "870fbdcfdac84bacae0ad22dba25f626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9964c1c76dd14f7dbce766b0c022ab51" is_pre_election: true
I20240501 14:04:03.733901 30508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8544c28e5c1448e8a069c1842d6638d9" candidate_uuid: "870fbdcfdac84bacae0ad22dba25f626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8883498098f4454bbfe563fb83d31899" is_pre_election: true
I20240501 14:04:03.734112 31032 raft_consensus.cc:2454] T 8544c28e5c1448e8a069c1842d6638d9 P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 870fbdcfdac84bacae0ad22dba25f626 in term 0.
I20240501 14:04:03.734133 30508 raft_consensus.cc:2454] T 8544c28e5c1448e8a069c1842d6638d9 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 870fbdcfdac84bacae0ad22dba25f626 in term 0.
I20240501 14:04:03.734558 30704 leader_election.cc:302] T 8544c28e5c1448e8a069c1842d6638d9 P 870fbdcfdac84bacae0ad22dba25f626 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 870fbdcfdac84bacae0ad22dba25f626, 9964c1c76dd14f7dbce766b0c022ab51; no voters:
I20240501 14:04:03.734822 31199 raft_consensus.cc:2781] T 8544c28e5c1448e8a069c1842d6638d9 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:04:03.734921 31199 raft_consensus.cc:482] T 8544c28e5c1448e8a069c1842d6638d9 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:04:03.735013 31199 raft_consensus.cc:3037] T 8544c28e5c1448e8a069c1842d6638d9 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:03.737146 31199 raft_consensus.cc:504] T 8544c28e5c1448e8a069c1842d6638d9 P 870fbdcfdac84bacae0ad22dba25f626 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:04:03.737740 31199 leader_election.cc:288] T 8544c28e5c1448e8a069c1842d6638d9 P 870fbdcfdac84bacae0ad22dba25f626 [CANDIDATE]: Term 1 election: Requested vote from peers 9964c1c76dd14f7dbce766b0c022ab51 (127.29.124.134:38101), 8883498098f4454bbfe563fb83d31899 (127.29.124.130:43099)
I20240501 14:04:03.738212 30508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8544c28e5c1448e8a069c1842d6638d9" candidate_uuid: "870fbdcfdac84bacae0ad22dba25f626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8883498098f4454bbfe563fb83d31899"
I20240501 14:04:03.738224 31032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8544c28e5c1448e8a069c1842d6638d9" candidate_uuid: "870fbdcfdac84bacae0ad22dba25f626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9964c1c76dd14f7dbce766b0c022ab51"
I20240501 14:04:03.738423 30508 raft_consensus.cc:3037] T 8544c28e5c1448e8a069c1842d6638d9 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:03.738425 31032 raft_consensus.cc:3037] T 8544c28e5c1448e8a069c1842d6638d9 P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:03.740248 30508 raft_consensus.cc:2454] T 8544c28e5c1448e8a069c1842d6638d9 P 8883498098f4454bbfe563fb83d31899 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 870fbdcfdac84bacae0ad22dba25f626 in term 1.
I20240501 14:04:03.740325 31032 raft_consensus.cc:2454] T 8544c28e5c1448e8a069c1842d6638d9 P 9964c1c76dd14f7dbce766b0c022ab51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 870fbdcfdac84bacae0ad22dba25f626 in term 1.
I20240501 14:04:03.740661 30707 leader_election.cc:302] T 8544c28e5c1448e8a069c1842d6638d9 P 870fbdcfdac84bacae0ad22dba25f626 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 870fbdcfdac84bacae0ad22dba25f626, 8883498098f4454bbfe563fb83d31899; no voters:
I20240501 14:04:03.740957 31199 raft_consensus.cc:2781] T 8544c28e5c1448e8a069c1842d6638d9 P 870fbdcfdac84bacae0ad22dba25f626 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:04:03.741107 31199 raft_consensus.cc:686] T 8544c28e5c1448e8a069c1842d6638d9 P 870fbdcfdac84bacae0ad22dba25f626 [term 1 LEADER]: Becoming Leader. State: Replica: 870fbdcfdac84bacae0ad22dba25f626, State: Running, Role: LEADER
I20240501 14:04:03.741425 31199 consensus_queue.cc:237] T 8544c28e5c1448e8a069c1842d6638d9 P 870fbdcfdac84bacae0ad22dba25f626 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:04:03.744390 30228 catalog_manager.cc:5484] T 8544c28e5c1448e8a069c1842d6638d9 P 870fbdcfdac84bacae0ad22dba25f626 reported cstate change: term changed from 0 to 1, leader changed from <none> to 870fbdcfdac84bacae0ad22dba25f626 (127.29.124.132). New cstate: current_term: 1 leader_uuid: "870fbdcfdac84bacae0ad22dba25f626" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } health_report { overall_health: UNKNOWN } } }
I20240501 14:04:04.023463 31199 consensus_queue.cc:1035] T eada160f92e446468cb1a1c78ef4790a P 870fbdcfdac84bacae0ad22dba25f626 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:04:04.038790 31975 consensus_queue.cc:1035] T eada160f92e446468cb1a1c78ef4790a P 870fbdcfdac84bacae0ad22dba25f626 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:04:04.092196 31114 consensus_queue.cc:1035] T 850c9d8997af4db8b1d28707dbd517cb P a3e62ccad691413ea2554182f73418a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:04:04.097345 31114 consensus_queue.cc:1035] T 850c9d8997af4db8b1d28707dbd517cb P a3e62ccad691413ea2554182f73418a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:04:04.115482 31971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:04:04.115810 31971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:04:04.123366 31971 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
I20240501 14:04:04.260440 30508 raft_consensus.cc:1261] T 8544c28e5c1448e8a069c1842d6638d9 P 8883498098f4454bbfe563fb83d31899 [term 1 FOLLOWER]: Refusing update from remote peer 870fbdcfdac84bacae0ad22dba25f626: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:04:04.260450 31032 raft_consensus.cc:1261] T 8544c28e5c1448e8a069c1842d6638d9 P 9964c1c76dd14f7dbce766b0c022ab51 [term 1 FOLLOWER]: Refusing update from remote peer 870fbdcfdac84bacae0ad22dba25f626: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:04:04.261065 31975 consensus_queue.cc:1035] T 8544c28e5c1448e8a069c1842d6638d9 P 870fbdcfdac84bacae0ad22dba25f626 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:04:04.262462 31975 consensus_queue.cc:1035] T 8544c28e5c1448e8a069c1842d6638d9 P 870fbdcfdac84bacae0ad22dba25f626 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:04:04.312050 32015 mvcc.cc:205] Tried to move back new op lower bound from 7022887912484098048 to 7022887910366924800. Current Snapshot: MvccSnapshot[applied={T|T < 7022887912484098048}]
INSERT report
rows total: 2000
time total: 245.997 ms
time per row: 0.122998 ms
I20240501 14:04:04.509730 30233 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35960:
name: "rebalance_test_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:04:04.519439 30882 tablet_service.cc:1467] Processing CreateTablet for tablet d8f5779ed9624273b2203b03213e83aa (DEFAULT_TABLE table=rebalance_test_table_10 [id=a3d5422b9dbe4f91ac15ccbe9c162d4f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:04.519902 30882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8f5779ed9624273b2203b03213e83aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:04.520581 30750 tablet_service.cc:1467] Processing CreateTablet for tablet 7ff70a696ab64c66a0829a3b390b8c8d (DEFAULT_TABLE table=rebalance_test_table_10 [id=a3d5422b9dbe4f91ac15ccbe9c162d4f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:04.521034 30750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ff70a696ab64c66a0829a3b390b8c8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:04.521126 30620 tablet_service.cc:1467] Processing CreateTablet for tablet d8f5779ed9624273b2203b03213e83aa (DEFAULT_TABLE table=rebalance_test_table_10 [id=a3d5422b9dbe4f91ac15ccbe9c162d4f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:04.521632 30620 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8f5779ed9624273b2203b03213e83aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:04.521632 30619 tablet_service.cc:1467] Processing CreateTablet for tablet 3e1b9355a8334dbca7276e013cdfa8ff (DEFAULT_TABLE table=rebalance_test_table_10 [id=a3d5422b9dbe4f91ac15ccbe9c162d4f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:04.522006 30619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e1b9355a8334dbca7276e013cdfa8ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:04.525662 31012 tablet_service.cc:1467] Processing CreateTablet for tablet 7ff70a696ab64c66a0829a3b390b8c8d (DEFAULT_TABLE table=rebalance_test_table_10 [id=a3d5422b9dbe4f91ac15ccbe9c162d4f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:04.526049 31012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ff70a696ab64c66a0829a3b390b8c8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:04.526331 30358 tablet_service.cc:1467] Processing CreateTablet for tablet 3e1b9355a8334dbca7276e013cdfa8ff (DEFAULT_TABLE table=rebalance_test_table_10 [id=a3d5422b9dbe4f91ac15ccbe9c162d4f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:04.526726 30358 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e1b9355a8334dbca7276e013cdfa8ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:04.529140 30489 tablet_service.cc:1467] Processing CreateTablet for tablet 7ff70a696ab64c66a0829a3b390b8c8d (DEFAULT_TABLE table=rebalance_test_table_10 [id=a3d5422b9dbe4f91ac15ccbe9c162d4f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:04.529621 30489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ff70a696ab64c66a0829a3b390b8c8d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:04.533686 30357 tablet_service.cc:1467] Processing CreateTablet for tablet d8f5779ed9624273b2203b03213e83aa (DEFAULT_TABLE table=rebalance_test_table_10 [id=a3d5422b9dbe4f91ac15ccbe9c162d4f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:04.534137 30357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d8f5779ed9624273b2203b03213e83aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:04.535663 32039 tablet_bootstrap.cc:492] T d8f5779ed9624273b2203b03213e83aa P 99f6ca2d65d94b33b7a1d2341c24179a: Bootstrap starting.
I20240501 14:04:04.536881 32040 tablet_bootstrap.cc:492] T d8f5779ed9624273b2203b03213e83aa P a3e62ccad691413ea2554182f73418a3: Bootstrap starting.
I20240501 14:04:04.537230 32039 tablet_bootstrap.cc:654] T d8f5779ed9624273b2203b03213e83aa P 99f6ca2d65d94b33b7a1d2341c24179a: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:04.539279 32040 tablet_bootstrap.cc:654] T d8f5779ed9624273b2203b03213e83aa P a3e62ccad691413ea2554182f73418a3: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:04.543344 32043 tablet_bootstrap.cc:492] T 3e1b9355a8334dbca7276e013cdfa8ff P c0fad54e965647aaaa34b56674646cb4: Bootstrap starting.
I20240501 14:04:04.543941 30882 tablet_service.cc:1467] Processing CreateTablet for tablet 3e1b9355a8334dbca7276e013cdfa8ff (DEFAULT_TABLE table=rebalance_test_table_10 [id=a3d5422b9dbe4f91ac15ccbe9c162d4f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:04.544397 30882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e1b9355a8334dbca7276e013cdfa8ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:04.545536 32043 tablet_bootstrap.cc:654] T 3e1b9355a8334dbca7276e013cdfa8ff P c0fad54e965647aaaa34b56674646cb4: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:04.547433 32044 tablet_bootstrap.cc:492] T 7ff70a696ab64c66a0829a3b390b8c8d P 870fbdcfdac84bacae0ad22dba25f626: Bootstrap starting.
I20240501 14:04:04.549942 32044 tablet_bootstrap.cc:654] T 7ff70a696ab64c66a0829a3b390b8c8d P 870fbdcfdac84bacae0ad22dba25f626: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:04.558234 32048 tablet_bootstrap.cc:492] T 7ff70a696ab64c66a0829a3b390b8c8d P 9964c1c76dd14f7dbce766b0c022ab51: Bootstrap starting.
I20240501 14:04:04.561342 32048 tablet_bootstrap.cc:654] T 7ff70a696ab64c66a0829a3b390b8c8d P 9964c1c76dd14f7dbce766b0c022ab51: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:04.568073 32043 tablet_bootstrap.cc:492] T 3e1b9355a8334dbca7276e013cdfa8ff P c0fad54e965647aaaa34b56674646cb4: No bootstrap required, opened a new log
I20240501 14:04:04.568256 32043 ts_tablet_manager.cc:1367] T 3e1b9355a8334dbca7276e013cdfa8ff P c0fad54e965647aaaa34b56674646cb4: Time spent bootstrapping tablet: real 0.025s user 0.001s sys 0.005s
I20240501 14:04:04.569078 32040 tablet_bootstrap.cc:492] T d8f5779ed9624273b2203b03213e83aa P a3e62ccad691413ea2554182f73418a3: No bootstrap required, opened a new log
I20240501 14:04:04.568996 32043 raft_consensus.cc:348] T 3e1b9355a8334dbca7276e013cdfa8ff P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } }
I20240501 14:04:04.569203 32043 raft_consensus.cc:374] T 3e1b9355a8334dbca7276e013cdfa8ff P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:04.569239 32040 ts_tablet_manager.cc:1367] T d8f5779ed9624273b2203b03213e83aa P a3e62ccad691413ea2554182f73418a3: Time spent bootstrapping tablet: real 0.032s user 0.001s sys 0.018s
I20240501 14:04:04.569319 32043 raft_consensus.cc:729] T 3e1b9355a8334dbca7276e013cdfa8ff P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0fad54e965647aaaa34b56674646cb4, State: Initialized, Role: FOLLOWER
I20240501 14:04:04.569658 32043 consensus_queue.cc:260] T 3e1b9355a8334dbca7276e013cdfa8ff P c0fad54e965647aaaa34b56674646cb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } }
I20240501 14:04:04.570283 32043 ts_tablet_manager.cc:1398] T 3e1b9355a8334dbca7276e013cdfa8ff P c0fad54e965647aaaa34b56674646cb4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:04:04.570326 32040 raft_consensus.cc:348] T d8f5779ed9624273b2203b03213e83aa P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:04:04.570683 32040 raft_consensus.cc:374] T d8f5779ed9624273b2203b03213e83aa P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:04.570865 32040 raft_consensus.cc:729] T d8f5779ed9624273b2203b03213e83aa P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3e62ccad691413ea2554182f73418a3, State: Initialized, Role: FOLLOWER
I20240501 14:04:04.571225 32039 tablet_bootstrap.cc:492] T d8f5779ed9624273b2203b03213e83aa P 99f6ca2d65d94b33b7a1d2341c24179a: No bootstrap required, opened a new log
I20240501 14:04:04.571154 32040 consensus_queue.cc:260] T d8f5779ed9624273b2203b03213e83aa P a3e62ccad691413ea2554182f73418a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:04:04.571391 32039 ts_tablet_manager.cc:1367] T d8f5779ed9624273b2203b03213e83aa P 99f6ca2d65d94b33b7a1d2341c24179a: Time spent bootstrapping tablet: real 0.036s user 0.008s sys 0.008s
I20240501 14:04:04.572225 32039 raft_consensus.cc:348] T d8f5779ed9624273b2203b03213e83aa P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:04:04.572443 32039 raft_consensus.cc:374] T d8f5779ed9624273b2203b03213e83aa P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:04.572530 32039 raft_consensus.cc:729] T d8f5779ed9624273b2203b03213e83aa P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99f6ca2d65d94b33b7a1d2341c24179a, State: Initialized, Role: FOLLOWER
I20240501 14:04:04.572741 32039 consensus_queue.cc:260] T d8f5779ed9624273b2203b03213e83aa P 99f6ca2d65d94b33b7a1d2341c24179a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:04:04.573294 32039 ts_tablet_manager.cc:1398] T d8f5779ed9624273b2203b03213e83aa P 99f6ca2d65d94b33b7a1d2341c24179a: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:04:04.574247 32039 tablet_bootstrap.cc:492] T 3e1b9355a8334dbca7276e013cdfa8ff P 99f6ca2d65d94b33b7a1d2341c24179a: Bootstrap starting.
I20240501 14:04:04.574550 32040 ts_tablet_manager.cc:1398] T d8f5779ed9624273b2203b03213e83aa P a3e62ccad691413ea2554182f73418a3: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240501 14:04:04.574888 32040 tablet_bootstrap.cc:492] T 3e1b9355a8334dbca7276e013cdfa8ff P a3e62ccad691413ea2554182f73418a3: Bootstrap starting.
I20240501 14:04:04.575904 32039 tablet_bootstrap.cc:654] T 3e1b9355a8334dbca7276e013cdfa8ff P 99f6ca2d65d94b33b7a1d2341c24179a: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:04.576758 32040 tablet_bootstrap.cc:654] T 3e1b9355a8334dbca7276e013cdfa8ff P a3e62ccad691413ea2554182f73418a3: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:04.577319 32049 tablet_bootstrap.cc:492] T 7ff70a696ab64c66a0829a3b390b8c8d P 8883498098f4454bbfe563fb83d31899: Bootstrap starting.
I20240501 14:04:04.577666 32043 tablet_bootstrap.cc:492] T d8f5779ed9624273b2203b03213e83aa P c0fad54e965647aaaa34b56674646cb4: Bootstrap starting.
I20240501 14:04:04.581116 32049 tablet_bootstrap.cc:654] T 7ff70a696ab64c66a0829a3b390b8c8d P 8883498098f4454bbfe563fb83d31899: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:04.582494 32043 tablet_bootstrap.cc:654] T d8f5779ed9624273b2203b03213e83aa P c0fad54e965647aaaa34b56674646cb4: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:04.582542 32048 tablet_bootstrap.cc:492] T 7ff70a696ab64c66a0829a3b390b8c8d P 9964c1c76dd14f7dbce766b0c022ab51: No bootstrap required, opened a new log
I20240501 14:04:04.582554 32044 tablet_bootstrap.cc:492] T 7ff70a696ab64c66a0829a3b390b8c8d P 870fbdcfdac84bacae0ad22dba25f626: No bootstrap required, opened a new log
I20240501 14:04:04.582712 32044 ts_tablet_manager.cc:1367] T 7ff70a696ab64c66a0829a3b390b8c8d P 870fbdcfdac84bacae0ad22dba25f626: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.003s
I20240501 14:04:04.582762 32048 ts_tablet_manager.cc:1367] T 7ff70a696ab64c66a0829a3b390b8c8d P 9964c1c76dd14f7dbce766b0c022ab51: Time spent bootstrapping tablet: real 0.025s user 0.006s sys 0.000s
I20240501 14:04:04.583590 32048 raft_consensus.cc:348] T 7ff70a696ab64c66a0829a3b390b8c8d P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:04:04.583876 32048 raft_consensus.cc:374] T 7ff70a696ab64c66a0829a3b390b8c8d P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:04.583838 32044 raft_consensus.cc:348] T 7ff70a696ab64c66a0829a3b390b8c8d P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:04:04.584074 32048 raft_consensus.cc:729] T 7ff70a696ab64c66a0829a3b390b8c8d P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9964c1c76dd14f7dbce766b0c022ab51, State: Initialized, Role: FOLLOWER
I20240501 14:04:04.584272 32044 raft_consensus.cc:374] T 7ff70a696ab64c66a0829a3b390b8c8d P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:04.584318 32048 consensus_queue.cc:260] T 7ff70a696ab64c66a0829a3b390b8c8d P 9964c1c76dd14f7dbce766b0c022ab51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:04:04.584429 32044 raft_consensus.cc:729] T 7ff70a696ab64c66a0829a3b390b8c8d P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 870fbdcfdac84bacae0ad22dba25f626, State: Initialized, Role: FOLLOWER
I20240501 14:04:04.584988 32044 consensus_queue.cc:260] T 7ff70a696ab64c66a0829a3b390b8c8d P 870fbdcfdac84bacae0ad22dba25f626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:04:04.585669 32048 ts_tablet_manager.cc:1398] T 7ff70a696ab64c66a0829a3b390b8c8d P 9964c1c76dd14f7dbce766b0c022ab51: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:04:04.585691 32044 ts_tablet_manager.cc:1398] T 7ff70a696ab64c66a0829a3b390b8c8d P 870fbdcfdac84bacae0ad22dba25f626: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:04:04.586587 32039 tablet_bootstrap.cc:492] T 3e1b9355a8334dbca7276e013cdfa8ff P 99f6ca2d65d94b33b7a1d2341c24179a: No bootstrap required, opened a new log
I20240501 14:04:04.586922 32039 ts_tablet_manager.cc:1367] T 3e1b9355a8334dbca7276e013cdfa8ff P 99f6ca2d65d94b33b7a1d2341c24179a: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
I20240501 14:04:04.588388 32039 raft_consensus.cc:348] T 3e1b9355a8334dbca7276e013cdfa8ff P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } }
I20240501 14:04:04.588560 32039 raft_consensus.cc:374] T 3e1b9355a8334dbca7276e013cdfa8ff P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:04.588634 32039 raft_consensus.cc:729] T 3e1b9355a8334dbca7276e013cdfa8ff P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99f6ca2d65d94b33b7a1d2341c24179a, State: Initialized, Role: FOLLOWER
I20240501 14:04:04.588851 32039 consensus_queue.cc:260] T 3e1b9355a8334dbca7276e013cdfa8ff P 99f6ca2d65d94b33b7a1d2341c24179a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } }
I20240501 14:04:04.589160 32043 tablet_bootstrap.cc:492] T d8f5779ed9624273b2203b03213e83aa P c0fad54e965647aaaa34b56674646cb4: No bootstrap required, opened a new log
I20240501 14:04:04.589417 32039 ts_tablet_manager.cc:1398] T 3e1b9355a8334dbca7276e013cdfa8ff P 99f6ca2d65d94b33b7a1d2341c24179a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:04:04.589416 32043 ts_tablet_manager.cc:1367] T d8f5779ed9624273b2203b03213e83aa P c0fad54e965647aaaa34b56674646cb4: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.001s
I20240501 14:04:04.590991 32040 tablet_bootstrap.cc:492] T 3e1b9355a8334dbca7276e013cdfa8ff P a3e62ccad691413ea2554182f73418a3: No bootstrap required, opened a new log
I20240501 14:04:04.591194 32040 ts_tablet_manager.cc:1367] T 3e1b9355a8334dbca7276e013cdfa8ff P a3e62ccad691413ea2554182f73418a3: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.005s
I20240501 14:04:04.591096 32043 raft_consensus.cc:348] T d8f5779ed9624273b2203b03213e83aa P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:04:04.591327 32043 raft_consensus.cc:374] T d8f5779ed9624273b2203b03213e83aa P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:04.591419 32043 raft_consensus.cc:729] T d8f5779ed9624273b2203b03213e83aa P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0fad54e965647aaaa34b56674646cb4, State: Initialized, Role: FOLLOWER
I20240501 14:04:04.591653 32043 consensus_queue.cc:260] T d8f5779ed9624273b2203b03213e83aa P c0fad54e965647aaaa34b56674646cb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:04:04.591871 32040 raft_consensus.cc:348] T 3e1b9355a8334dbca7276e013cdfa8ff P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } }
I20240501 14:04:04.592121 32040 raft_consensus.cc:374] T 3e1b9355a8334dbca7276e013cdfa8ff P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:04.592218 32040 raft_consensus.cc:729] T 3e1b9355a8334dbca7276e013cdfa8ff P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3e62ccad691413ea2554182f73418a3, State: Initialized, Role: FOLLOWER
I20240501 14:04:04.592259 32043 ts_tablet_manager.cc:1398] T d8f5779ed9624273b2203b03213e83aa P c0fad54e965647aaaa34b56674646cb4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:04:04.592485 32040 consensus_queue.cc:260] T 3e1b9355a8334dbca7276e013cdfa8ff P a3e62ccad691413ea2554182f73418a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } }
I20240501 14:04:04.593045 32040 ts_tablet_manager.cc:1398] T 3e1b9355a8334dbca7276e013cdfa8ff P a3e62ccad691413ea2554182f73418a3: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:04:04.593798 32049 tablet_bootstrap.cc:492] T 7ff70a696ab64c66a0829a3b390b8c8d P 8883498098f4454bbfe563fb83d31899: No bootstrap required, opened a new log
I20240501 14:04:04.594132 32049 ts_tablet_manager.cc:1367] T 7ff70a696ab64c66a0829a3b390b8c8d P 8883498098f4454bbfe563fb83d31899: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.006s
I20240501 14:04:04.594935 32049 raft_consensus.cc:348] T 7ff70a696ab64c66a0829a3b390b8c8d P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:04:04.595160 32049 raft_consensus.cc:374] T 7ff70a696ab64c66a0829a3b390b8c8d P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:04.595260 32049 raft_consensus.cc:729] T 7ff70a696ab64c66a0829a3b390b8c8d P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8883498098f4454bbfe563fb83d31899, State: Initialized, Role: FOLLOWER
I20240501 14:04:04.595494 32049 consensus_queue.cc:260] T 7ff70a696ab64c66a0829a3b390b8c8d P 8883498098f4454bbfe563fb83d31899 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:04:04.596027 32049 ts_tablet_manager.cc:1398] T 7ff70a696ab64c66a0829a3b390b8c8d P 8883498098f4454bbfe563fb83d31899: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:04:04.604334 32020 raft_consensus.cc:482] T d8f5779ed9624273b2203b03213e83aa P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:04:04.604476 32020 raft_consensus.cc:504] T d8f5779ed9624273b2203b03213e83aa P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:04:04.605077 32020 leader_election.cc:288] T d8f5779ed9624273b2203b03213e83aa P a3e62ccad691413ea2554182f73418a3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99f6ca2d65d94b33b7a1d2341c24179a (127.29.124.133:35195), c0fad54e965647aaaa34b56674646cb4 (127.29.124.129:42363)
I20240501 14:04:04.605511 30901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8f5779ed9624273b2203b03213e83aa" candidate_uuid: "a3e62ccad691413ea2554182f73418a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" is_pre_election: true
I20240501 14:04:04.605513 30378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8f5779ed9624273b2203b03213e83aa" candidate_uuid: "a3e62ccad691413ea2554182f73418a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0fad54e965647aaaa34b56674646cb4" is_pre_election: true
I20240501 14:04:04.605695 30901 raft_consensus.cc:2454] T d8f5779ed9624273b2203b03213e83aa P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3e62ccad691413ea2554182f73418a3 in term 0.
I20240501 14:04:04.605746 30378 raft_consensus.cc:2454] T d8f5779ed9624273b2203b03213e83aa P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a3e62ccad691413ea2554182f73418a3 in term 0.
I20240501 14:04:04.606076 30575 leader_election.cc:302] T d8f5779ed9624273b2203b03213e83aa P a3e62ccad691413ea2554182f73418a3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99f6ca2d65d94b33b7a1d2341c24179a, a3e62ccad691413ea2554182f73418a3; no voters:
I20240501 14:04:04.606331 32020 raft_consensus.cc:2781] T d8f5779ed9624273b2203b03213e83aa P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:04:04.606428 32020 raft_consensus.cc:482] T d8f5779ed9624273b2203b03213e83aa P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:04:04.606492 32020 raft_consensus.cc:3037] T d8f5779ed9624273b2203b03213e83aa P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:04.608417 32020 raft_consensus.cc:504] T d8f5779ed9624273b2203b03213e83aa P a3e62ccad691413ea2554182f73418a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:04:04.608989 32020 leader_election.cc:288] T d8f5779ed9624273b2203b03213e83aa P a3e62ccad691413ea2554182f73418a3 [CANDIDATE]: Term 1 election: Requested vote from peers 99f6ca2d65d94b33b7a1d2341c24179a (127.29.124.133:35195), c0fad54e965647aaaa34b56674646cb4 (127.29.124.129:42363)
I20240501 14:04:04.609411 30901 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8f5779ed9624273b2203b03213e83aa" candidate_uuid: "a3e62ccad691413ea2554182f73418a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99f6ca2d65d94b33b7a1d2341c24179a"
I20240501 14:04:04.609442 30378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d8f5779ed9624273b2203b03213e83aa" candidate_uuid: "a3e62ccad691413ea2554182f73418a3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0fad54e965647aaaa34b56674646cb4"
I20240501 14:04:04.609580 30901 raft_consensus.cc:3037] T d8f5779ed9624273b2203b03213e83aa P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:04.609668 30378 raft_consensus.cc:3037] T d8f5779ed9624273b2203b03213e83aa P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:04.611413 30901 raft_consensus.cc:2454] T d8f5779ed9624273b2203b03213e83aa P 99f6ca2d65d94b33b7a1d2341c24179a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3e62ccad691413ea2554182f73418a3 in term 1.
I20240501 14:04:04.611646 30378 raft_consensus.cc:2454] T d8f5779ed9624273b2203b03213e83aa P c0fad54e965647aaaa34b56674646cb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a3e62ccad691413ea2554182f73418a3 in term 1.
I20240501 14:04:04.611825 30575 leader_election.cc:302] T d8f5779ed9624273b2203b03213e83aa P a3e62ccad691413ea2554182f73418a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99f6ca2d65d94b33b7a1d2341c24179a, a3e62ccad691413ea2554182f73418a3; no voters:
I20240501 14:04:04.612107 32020 raft_consensus.cc:2781] T d8f5779ed9624273b2203b03213e83aa P a3e62ccad691413ea2554182f73418a3 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:04:04.612228 32020 raft_consensus.cc:686] T d8f5779ed9624273b2203b03213e83aa P a3e62ccad691413ea2554182f73418a3 [term 1 LEADER]: Becoming Leader. State: Replica: a3e62ccad691413ea2554182f73418a3, State: Running, Role: LEADER
I20240501 14:04:04.612505 32020 consensus_queue.cc:237] T d8f5779ed9624273b2203b03213e83aa P a3e62ccad691413ea2554182f73418a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:04:04.615562 30236 catalog_manager.cc:5484] T d8f5779ed9624273b2203b03213e83aa P a3e62ccad691413ea2554182f73418a3 reported cstate change: term changed from 0 to 1, leader changed from <none> to a3e62ccad691413ea2554182f73418a3 (127.29.124.131). New cstate: current_term: 1 leader_uuid: "a3e62ccad691413ea2554182f73418a3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } health_report { overall_health: HEALTHY } } }
I20240501 14:04:04.728320 32023 raft_consensus.cc:482] T 7ff70a696ab64c66a0829a3b390b8c8d P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:04:04.728534 32023 raft_consensus.cc:504] T 7ff70a696ab64c66a0829a3b390b8c8d P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:04:04.729226 32023 leader_election.cc:288] T 7ff70a696ab64c66a0829a3b390b8c8d P 8883498098f4454bbfe563fb83d31899 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9964c1c76dd14f7dbce766b0c022ab51 (127.29.124.134:38101), 870fbdcfdac84bacae0ad22dba25f626 (127.29.124.132:44267)
I20240501 14:04:04.729771 31032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ff70a696ab64c66a0829a3b390b8c8d" candidate_uuid: "8883498098f4454bbfe563fb83d31899" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9964c1c76dd14f7dbce766b0c022ab51" is_pre_election: true
I20240501 14:04:04.729816 30771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ff70a696ab64c66a0829a3b390b8c8d" candidate_uuid: "8883498098f4454bbfe563fb83d31899" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "870fbdcfdac84bacae0ad22dba25f626" is_pre_election: true
I20240501 14:04:04.729974 31032 raft_consensus.cc:2454] T 7ff70a696ab64c66a0829a3b390b8c8d P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8883498098f4454bbfe563fb83d31899 in term 0.
I20240501 14:04:04.730022 30771 raft_consensus.cc:2454] T 7ff70a696ab64c66a0829a3b390b8c8d P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8883498098f4454bbfe563fb83d31899 in term 0.
I20240501 14:04:04.730437 30442 leader_election.cc:302] T 7ff70a696ab64c66a0829a3b390b8c8d P 8883498098f4454bbfe563fb83d31899 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8883498098f4454bbfe563fb83d31899, 9964c1c76dd14f7dbce766b0c022ab51; no voters:
I20240501 14:04:04.730731 32023 raft_consensus.cc:2781] T 7ff70a696ab64c66a0829a3b390b8c8d P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:04:04.730841 32023 raft_consensus.cc:482] T 7ff70a696ab64c66a0829a3b390b8c8d P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:04:04.730926 32023 raft_consensus.cc:3037] T 7ff70a696ab64c66a0829a3b390b8c8d P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:04.733006 32023 raft_consensus.cc:504] T 7ff70a696ab64c66a0829a3b390b8c8d P 8883498098f4454bbfe563fb83d31899 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:04:04.733630 32023 leader_election.cc:288] T 7ff70a696ab64c66a0829a3b390b8c8d P 8883498098f4454bbfe563fb83d31899 [CANDIDATE]: Term 1 election: Requested vote from peers 9964c1c76dd14f7dbce766b0c022ab51 (127.29.124.134:38101), 870fbdcfdac84bacae0ad22dba25f626 (127.29.124.132:44267)
I20240501 14:04:04.734035 31032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ff70a696ab64c66a0829a3b390b8c8d" candidate_uuid: "8883498098f4454bbfe563fb83d31899" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9964c1c76dd14f7dbce766b0c022ab51"
I20240501 14:04:04.734117 30771 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ff70a696ab64c66a0829a3b390b8c8d" candidate_uuid: "8883498098f4454bbfe563fb83d31899" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "870fbdcfdac84bacae0ad22dba25f626"
I20240501 14:04:04.734295 30771 raft_consensus.cc:3037] T 7ff70a696ab64c66a0829a3b390b8c8d P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:04.734292 31032 raft_consensus.cc:3037] T 7ff70a696ab64c66a0829a3b390b8c8d P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:04.736999 31032 raft_consensus.cc:2454] T 7ff70a696ab64c66a0829a3b390b8c8d P 9964c1c76dd14f7dbce766b0c022ab51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8883498098f4454bbfe563fb83d31899 in term 1.
I20240501 14:04:04.736992 30771 raft_consensus.cc:2454] T 7ff70a696ab64c66a0829a3b390b8c8d P 870fbdcfdac84bacae0ad22dba25f626 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8883498098f4454bbfe563fb83d31899 in term 1.
I20240501 14:04:04.737411 30445 leader_election.cc:302] T 7ff70a696ab64c66a0829a3b390b8c8d P 8883498098f4454bbfe563fb83d31899 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 870fbdcfdac84bacae0ad22dba25f626, 8883498098f4454bbfe563fb83d31899; no voters:
I20240501 14:04:04.737757 32023 raft_consensus.cc:2781] T 7ff70a696ab64c66a0829a3b390b8c8d P 8883498098f4454bbfe563fb83d31899 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:04:04.737896 32023 raft_consensus.cc:686] T 7ff70a696ab64c66a0829a3b390b8c8d P 8883498098f4454bbfe563fb83d31899 [term 1 LEADER]: Becoming Leader. State: Replica: 8883498098f4454bbfe563fb83d31899, State: Running, Role: LEADER
I20240501 14:04:04.738209 32023 consensus_queue.cc:237] T 7ff70a696ab64c66a0829a3b390b8c8d P 8883498098f4454bbfe563fb83d31899 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:04:04.741681 30236 catalog_manager.cc:5484] T 7ff70a696ab64c66a0829a3b390b8c8d P 8883498098f4454bbfe563fb83d31899 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8883498098f4454bbfe563fb83d31899 (127.29.124.130). New cstate: current_term: 1 leader_uuid: "8883498098f4454bbfe563fb83d31899" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } health_report { overall_health: UNKNOWN } } }
I20240501 14:04:04.759456 31896 raft_consensus.cc:482] T 3e1b9355a8334dbca7276e013cdfa8ff P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:04:04.759618 31896 raft_consensus.cc:504] T 3e1b9355a8334dbca7276e013cdfa8ff P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } }
I20240501 14:04:04.760260 31896 leader_election.cc:288] T 3e1b9355a8334dbca7276e013cdfa8ff P 99f6ca2d65d94b33b7a1d2341c24179a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a3e62ccad691413ea2554182f73418a3 (127.29.124.131:43965), c0fad54e965647aaaa34b56674646cb4 (127.29.124.129:42363)
I20240501 14:04:04.760699 30638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1b9355a8334dbca7276e013cdfa8ff" candidate_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3e62ccad691413ea2554182f73418a3" is_pre_election: true
I20240501 14:04:04.760740 30378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1b9355a8334dbca7276e013cdfa8ff" candidate_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0fad54e965647aaaa34b56674646cb4" is_pre_election: true
I20240501 14:04:04.760890 30638 raft_consensus.cc:2454] T 3e1b9355a8334dbca7276e013cdfa8ff P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99f6ca2d65d94b33b7a1d2341c24179a in term 0.
I20240501 14:04:04.760931 30378 raft_consensus.cc:2454] T 3e1b9355a8334dbca7276e013cdfa8ff P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99f6ca2d65d94b33b7a1d2341c24179a in term 0.
I20240501 14:04:04.761349 30835 leader_election.cc:302] T 3e1b9355a8334dbca7276e013cdfa8ff P 99f6ca2d65d94b33b7a1d2341c24179a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99f6ca2d65d94b33b7a1d2341c24179a, a3e62ccad691413ea2554182f73418a3; no voters:
I20240501 14:04:04.761606 31896 raft_consensus.cc:2781] T 3e1b9355a8334dbca7276e013cdfa8ff P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:04:04.761714 31896 raft_consensus.cc:482] T 3e1b9355a8334dbca7276e013cdfa8ff P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:04:04.761781 31896 raft_consensus.cc:3037] T 3e1b9355a8334dbca7276e013cdfa8ff P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:04.763568 31896 raft_consensus.cc:504] T 3e1b9355a8334dbca7276e013cdfa8ff P 99f6ca2d65d94b33b7a1d2341c24179a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } }
I20240501 14:04:04.764184 31896 leader_election.cc:288] T 3e1b9355a8334dbca7276e013cdfa8ff P 99f6ca2d65d94b33b7a1d2341c24179a [CANDIDATE]: Term 1 election: Requested vote from peers a3e62ccad691413ea2554182f73418a3 (127.29.124.131:43965), c0fad54e965647aaaa34b56674646cb4 (127.29.124.129:42363)
I20240501 14:04:04.764602 30638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1b9355a8334dbca7276e013cdfa8ff" candidate_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3e62ccad691413ea2554182f73418a3"
I20240501 14:04:04.764703 30378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e1b9355a8334dbca7276e013cdfa8ff" candidate_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0fad54e965647aaaa34b56674646cb4"
I20240501 14:04:04.764781 30638 raft_consensus.cc:3037] T 3e1b9355a8334dbca7276e013cdfa8ff P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:04.764899 30378 raft_consensus.cc:3037] T 3e1b9355a8334dbca7276e013cdfa8ff P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:04.766784 30378 raft_consensus.cc:2454] T 3e1b9355a8334dbca7276e013cdfa8ff P c0fad54e965647aaaa34b56674646cb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99f6ca2d65d94b33b7a1d2341c24179a in term 1.
I20240501 14:04:04.766791 30638 raft_consensus.cc:2454] T 3e1b9355a8334dbca7276e013cdfa8ff P a3e62ccad691413ea2554182f73418a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99f6ca2d65d94b33b7a1d2341c24179a in term 1.
I20240501 14:04:04.767176 30835 leader_election.cc:302] T 3e1b9355a8334dbca7276e013cdfa8ff P 99f6ca2d65d94b33b7a1d2341c24179a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99f6ca2d65d94b33b7a1d2341c24179a, a3e62ccad691413ea2554182f73418a3; no voters:
I20240501 14:04:04.767493 31896 raft_consensus.cc:2781] T 3e1b9355a8334dbca7276e013cdfa8ff P 99f6ca2d65d94b33b7a1d2341c24179a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:04:04.767639 31896 raft_consensus.cc:686] T 3e1b9355a8334dbca7276e013cdfa8ff P 99f6ca2d65d94b33b7a1d2341c24179a [term 1 LEADER]: Becoming Leader. State: Replica: 99f6ca2d65d94b33b7a1d2341c24179a, State: Running, Role: LEADER
I20240501 14:04:04.767976 31896 consensus_queue.cc:237] T 3e1b9355a8334dbca7276e013cdfa8ff P 99f6ca2d65d94b33b7a1d2341c24179a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } }
I20240501 14:04:04.771009 30236 catalog_manager.cc:5484] T 3e1b9355a8334dbca7276e013cdfa8ff P 99f6ca2d65d94b33b7a1d2341c24179a reported cstate change: term changed from 0 to 1, leader changed from <none> to 99f6ca2d65d94b33b7a1d2341c24179a (127.29.124.133). New cstate: current_term: 1 leader_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } health_report { overall_health: UNKNOWN } } }
I20240501 14:04:05.051609 31114 consensus_queue.cc:1035] T d8f5779ed9624273b2203b03213e83aa P a3e62ccad691413ea2554182f73418a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:04:05.057859 31114 consensus_queue.cc:1035] T d8f5779ed9624273b2203b03213e83aa P a3e62ccad691413ea2554182f73418a3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:04:05.131376 32058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:04:05.131716 32058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 14:04:05.138146 32059 consensus_queue.cc:1035] T 7ff70a696ab64c66a0829a3b390b8c8d P 8883498098f4454bbfe563fb83d31899 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:04:05.140022 32058 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:04:05.146167 31714 consensus_queue.cc:1035] T 7ff70a696ab64c66a0829a3b390b8c8d P 8883498098f4454bbfe563fb83d31899 [LEADER]: Connected to new peer: Peer: permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:04:05.155663 31896 consensus_queue.cc:1035] T 3e1b9355a8334dbca7276e013cdfa8ff P 99f6ca2d65d94b33b7a1d2341c24179a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:04:05.160297 31896 consensus_queue.cc:1035] T 3e1b9355a8334dbca7276e013cdfa8ff P 99f6ca2d65d94b33b7a1d2341c24179a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_10'
INSERT report
rows total: 2000
time total: 245.154 ms
time per row: 0.122577 ms
I20240501 14:04:05.539211 30232 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35960:
name: "rebalance_test_table_11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:04:05.547897 31012 tablet_service.cc:1467] Processing CreateTablet for tablet abf88dbfc38745a5a5947e425c01dfd8 (DEFAULT_TABLE table=rebalance_test_table_11 [id=af605242ed0e4cde8b09fa96673bcb66]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:05.548147 30489 tablet_service.cc:1467] Processing CreateTablet for tablet abf88dbfc38745a5a5947e425c01dfd8 (DEFAULT_TABLE table=rebalance_test_table_11 [id=af605242ed0e4cde8b09fa96673bcb66]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:05.548398 31012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abf88dbfc38745a5a5947e425c01dfd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:05.548381 30750 tablet_service.cc:1467] Processing CreateTablet for tablet abf88dbfc38745a5a5947e425c01dfd8 (DEFAULT_TABLE table=rebalance_test_table_11 [id=af605242ed0e4cde8b09fa96673bcb66]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:05.548549 30489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abf88dbfc38745a5a5947e425c01dfd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:05.548810 30750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet abf88dbfc38745a5a5947e425c01dfd8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:05.550109 30882 tablet_service.cc:1467] Processing CreateTablet for tablet 0e2a5822d4494ea5b733120bc14c09ea (DEFAULT_TABLE table=rebalance_test_table_11 [id=af605242ed0e4cde8b09fa96673bcb66]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:05.550537 30882 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e2a5822d4494ea5b733120bc14c09ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:05.551501 30488 tablet_service.cc:1467] Processing CreateTablet for tablet 3200e86d665a497b871b42ac4c2778f1 (DEFAULT_TABLE table=rebalance_test_table_11 [id=af605242ed0e4cde8b09fa96673bcb66]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:05.551870 30488 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3200e86d665a497b871b42ac4c2778f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:05.551988 30357 tablet_service.cc:1467] Processing CreateTablet for tablet 0e2a5822d4494ea5b733120bc14c09ea (DEFAULT_TABLE table=rebalance_test_table_11 [id=af605242ed0e4cde8b09fa96673bcb66]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:05.552371 30357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e2a5822d4494ea5b733120bc14c09ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:05.553690 30619 tablet_service.cc:1467] Processing CreateTablet for tablet 0e2a5822d4494ea5b733120bc14c09ea (DEFAULT_TABLE table=rebalance_test_table_11 [id=af605242ed0e4cde8b09fa96673bcb66]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:05.554116 30619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0e2a5822d4494ea5b733120bc14c09ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:05.557668 30751 tablet_service.cc:1467] Processing CreateTablet for tablet 3200e86d665a497b871b42ac4c2778f1 (DEFAULT_TABLE table=rebalance_test_table_11 [id=af605242ed0e4cde8b09fa96673bcb66]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:05.558099 30751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3200e86d665a497b871b42ac4c2778f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:05.562425 31013 tablet_service.cc:1467] Processing CreateTablet for tablet 3200e86d665a497b871b42ac4c2778f1 (DEFAULT_TABLE table=rebalance_test_table_11 [id=af605242ed0e4cde8b09fa96673bcb66]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:05.562930 31013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3200e86d665a497b871b42ac4c2778f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:05.579893 32127 tablet_bootstrap.cc:492] T 0e2a5822d4494ea5b733120bc14c09ea P a3e62ccad691413ea2554182f73418a3: Bootstrap starting.
I20240501 14:04:05.582077 32127 tablet_bootstrap.cc:654] T 0e2a5822d4494ea5b733120bc14c09ea P a3e62ccad691413ea2554182f73418a3: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:05.582446 32126 tablet_bootstrap.cc:492] T 0e2a5822d4494ea5b733120bc14c09ea P 99f6ca2d65d94b33b7a1d2341c24179a: Bootstrap starting.
I20240501 14:04:05.582866 32124 tablet_bootstrap.cc:492] T abf88dbfc38745a5a5947e425c01dfd8 P 870fbdcfdac84bacae0ad22dba25f626: Bootstrap starting.
I20240501 14:04:05.583602 32125 tablet_bootstrap.cc:492] T 3200e86d665a497b871b42ac4c2778f1 P 9964c1c76dd14f7dbce766b0c022ab51: Bootstrap starting.
I20240501 14:04:05.584928 32124 tablet_bootstrap.cc:654] T abf88dbfc38745a5a5947e425c01dfd8 P 870fbdcfdac84bacae0ad22dba25f626: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:05.587002 32122 tablet_bootstrap.cc:492] T abf88dbfc38745a5a5947e425c01dfd8 P 8883498098f4454bbfe563fb83d31899: Bootstrap starting.
I20240501 14:04:05.587494 32126 tablet_bootstrap.cc:654] T 0e2a5822d4494ea5b733120bc14c09ea P 99f6ca2d65d94b33b7a1d2341c24179a: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:05.588443 32125 tablet_bootstrap.cc:654] T 3200e86d665a497b871b42ac4c2778f1 P 9964c1c76dd14f7dbce766b0c022ab51: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:05.589028 32122 tablet_bootstrap.cc:654] T abf88dbfc38745a5a5947e425c01dfd8 P 8883498098f4454bbfe563fb83d31899: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:05.590811 32124 tablet_bootstrap.cc:492] T abf88dbfc38745a5a5947e425c01dfd8 P 870fbdcfdac84bacae0ad22dba25f626: No bootstrap required, opened a new log
I20240501 14:04:05.591048 32124 ts_tablet_manager.cc:1367] T abf88dbfc38745a5a5947e425c01dfd8 P 870fbdcfdac84bacae0ad22dba25f626: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240501 14:04:05.591976 32124 raft_consensus.cc:348] T abf88dbfc38745a5a5947e425c01dfd8 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:04:05.592360 32124 raft_consensus.cc:374] T abf88dbfc38745a5a5947e425c01dfd8 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:05.592502 32124 raft_consensus.cc:729] T abf88dbfc38745a5a5947e425c01dfd8 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 870fbdcfdac84bacae0ad22dba25f626, State: Initialized, Role: FOLLOWER
I20240501 14:04:05.592829 32124 consensus_queue.cc:260] T abf88dbfc38745a5a5947e425c01dfd8 P 870fbdcfdac84bacae0ad22dba25f626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:04:05.594383 32124 ts_tablet_manager.cc:1398] T abf88dbfc38745a5a5947e425c01dfd8 P 870fbdcfdac84bacae0ad22dba25f626: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:04:05.594698 32124 tablet_bootstrap.cc:492] T 3200e86d665a497b871b42ac4c2778f1 P 870fbdcfdac84bacae0ad22dba25f626: Bootstrap starting.
I20240501 14:04:05.594931 32127 tablet_bootstrap.cc:492] T 0e2a5822d4494ea5b733120bc14c09ea P a3e62ccad691413ea2554182f73418a3: No bootstrap required, opened a new log
I20240501 14:04:05.595171 32127 ts_tablet_manager.cc:1367] T 0e2a5822d4494ea5b733120bc14c09ea P a3e62ccad691413ea2554182f73418a3: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.005s
I20240501 14:04:05.596015 32127 raft_consensus.cc:348] T 0e2a5822d4494ea5b733120bc14c09ea P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:04:05.596273 32124 tablet_bootstrap.cc:654] T 3200e86d665a497b871b42ac4c2778f1 P 870fbdcfdac84bacae0ad22dba25f626: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:05.596415 32127 raft_consensus.cc:374] T 0e2a5822d4494ea5b733120bc14c09ea P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:05.596549 32127 raft_consensus.cc:729] T 0e2a5822d4494ea5b733120bc14c09ea P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3e62ccad691413ea2554182f73418a3, State: Initialized, Role: FOLLOWER
I20240501 14:04:05.596885 32127 consensus_queue.cc:260] T 0e2a5822d4494ea5b733120bc14c09ea P a3e62ccad691413ea2554182f73418a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:04:05.597640 32127 ts_tablet_manager.cc:1398] T 0e2a5822d4494ea5b733120bc14c09ea P a3e62ccad691413ea2554182f73418a3: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:04:05.599874 32122 tablet_bootstrap.cc:492] T abf88dbfc38745a5a5947e425c01dfd8 P 8883498098f4454bbfe563fb83d31899: No bootstrap required, opened a new log
I20240501 14:04:05.600463 32123 tablet_bootstrap.cc:492] T 0e2a5822d4494ea5b733120bc14c09ea P c0fad54e965647aaaa34b56674646cb4: Bootstrap starting.
I20240501 14:04:05.601011 32122 ts_tablet_manager.cc:1367] T abf88dbfc38745a5a5947e425c01dfd8 P 8883498098f4454bbfe563fb83d31899: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.004s
I20240501 14:04:05.601965 32124 tablet_bootstrap.cc:492] T 3200e86d665a497b871b42ac4c2778f1 P 870fbdcfdac84bacae0ad22dba25f626: No bootstrap required, opened a new log
I20240501 14:04:05.602142 32124 ts_tablet_manager.cc:1367] T 3200e86d665a497b871b42ac4c2778f1 P 870fbdcfdac84bacae0ad22dba25f626: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240501 14:04:05.601925 32122 raft_consensus.cc:348] T abf88dbfc38745a5a5947e425c01dfd8 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:04:05.602656 32122 raft_consensus.cc:374] T abf88dbfc38745a5a5947e425c01dfd8 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:05.602850 32124 raft_consensus.cc:348] T 3200e86d665a497b871b42ac4c2778f1 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:04:05.603060 32124 raft_consensus.cc:374] T 3200e86d665a497b871b42ac4c2778f1 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:05.602800 32122 raft_consensus.cc:729] T abf88dbfc38745a5a5947e425c01dfd8 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8883498098f4454bbfe563fb83d31899, State: Initialized, Role: FOLLOWER
I20240501 14:04:05.603137 32124 raft_consensus.cc:729] T 3200e86d665a497b871b42ac4c2778f1 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 870fbdcfdac84bacae0ad22dba25f626, State: Initialized, Role: FOLLOWER
I20240501 14:04:05.603394 32124 consensus_queue.cc:260] T 3200e86d665a497b871b42ac4c2778f1 P 870fbdcfdac84bacae0ad22dba25f626 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:04:05.603401 32122 consensus_queue.cc:260] T abf88dbfc38745a5a5947e425c01dfd8 P 8883498098f4454bbfe563fb83d31899 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:04:05.604061 32122 ts_tablet_manager.cc:1398] T abf88dbfc38745a5a5947e425c01dfd8 P 8883498098f4454bbfe563fb83d31899: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:04:05.604326 32124 ts_tablet_manager.cc:1398] T 3200e86d665a497b871b42ac4c2778f1 P 870fbdcfdac84bacae0ad22dba25f626: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:04:05.605455 32126 tablet_bootstrap.cc:492] T 0e2a5822d4494ea5b733120bc14c09ea P 99f6ca2d65d94b33b7a1d2341c24179a: No bootstrap required, opened a new log
I20240501 14:04:05.605547 32122 tablet_bootstrap.cc:492] T 3200e86d665a497b871b42ac4c2778f1 P 8883498098f4454bbfe563fb83d31899: Bootstrap starting.
I20240501 14:04:05.605679 32126 ts_tablet_manager.cc:1367] T 0e2a5822d4494ea5b733120bc14c09ea P 99f6ca2d65d94b33b7a1d2341c24179a: Time spent bootstrapping tablet: real 0.023s user 0.008s sys 0.000s
I20240501 14:04:05.606135 32123 tablet_bootstrap.cc:654] T 0e2a5822d4494ea5b733120bc14c09ea P c0fad54e965647aaaa34b56674646cb4: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:05.607189 32122 tablet_bootstrap.cc:654] T 3200e86d665a497b871b42ac4c2778f1 P 8883498098f4454bbfe563fb83d31899: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:05.607740 32126 raft_consensus.cc:348] T 0e2a5822d4494ea5b733120bc14c09ea P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:04:05.608016 32126 raft_consensus.cc:374] T 0e2a5822d4494ea5b733120bc14c09ea P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:05.608148 32126 raft_consensus.cc:729] T 0e2a5822d4494ea5b733120bc14c09ea P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99f6ca2d65d94b33b7a1d2341c24179a, State: Initialized, Role: FOLLOWER
I20240501 14:04:05.608454 32126 consensus_queue.cc:260] T 0e2a5822d4494ea5b733120bc14c09ea P 99f6ca2d65d94b33b7a1d2341c24179a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:04:05.609161 32126 ts_tablet_manager.cc:1398] T 0e2a5822d4494ea5b733120bc14c09ea P 99f6ca2d65d94b33b7a1d2341c24179a: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:04:05.610301 32125 tablet_bootstrap.cc:492] T 3200e86d665a497b871b42ac4c2778f1 P 9964c1c76dd14f7dbce766b0c022ab51: No bootstrap required, opened a new log
I20240501 14:04:05.610512 32125 ts_tablet_manager.cc:1367] T 3200e86d665a497b871b42ac4c2778f1 P 9964c1c76dd14f7dbce766b0c022ab51: Time spent bootstrapping tablet: real 0.027s user 0.006s sys 0.006s
I20240501 14:04:05.613310 32125 raft_consensus.cc:348] T 3200e86d665a497b871b42ac4c2778f1 P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:04:05.613551 32125 raft_consensus.cc:374] T 3200e86d665a497b871b42ac4c2778f1 P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:05.613662 32125 raft_consensus.cc:729] T 3200e86d665a497b871b42ac4c2778f1 P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9964c1c76dd14f7dbce766b0c022ab51, State: Initialized, Role: FOLLOWER
I20240501 14:04:05.613938 32125 consensus_queue.cc:260] T 3200e86d665a497b871b42ac4c2778f1 P 9964c1c76dd14f7dbce766b0c022ab51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:04:05.614500 32125 ts_tablet_manager.cc:1398] T 3200e86d665a497b871b42ac4c2778f1 P 9964c1c76dd14f7dbce766b0c022ab51: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:04:05.615263 32125 tablet_bootstrap.cc:492] T abf88dbfc38745a5a5947e425c01dfd8 P 9964c1c76dd14f7dbce766b0c022ab51: Bootstrap starting.
I20240501 14:04:05.615906 32122 tablet_bootstrap.cc:492] T 3200e86d665a497b871b42ac4c2778f1 P 8883498098f4454bbfe563fb83d31899: No bootstrap required, opened a new log
I20240501 14:04:05.615979 32123 tablet_bootstrap.cc:492] T 0e2a5822d4494ea5b733120bc14c09ea P c0fad54e965647aaaa34b56674646cb4: No bootstrap required, opened a new log
I20240501 14:04:05.616147 32122 ts_tablet_manager.cc:1367] T 3200e86d665a497b871b42ac4c2778f1 P 8883498098f4454bbfe563fb83d31899: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.005s
I20240501 14:04:05.616205 32123 ts_tablet_manager.cc:1367] T 0e2a5822d4494ea5b733120bc14c09ea P c0fad54e965647aaaa34b56674646cb4: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.004s
I20240501 14:04:05.616834 32125 tablet_bootstrap.cc:654] T abf88dbfc38745a5a5947e425c01dfd8 P 9964c1c76dd14f7dbce766b0c022ab51: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:05.616972 32122 raft_consensus.cc:348] T 3200e86d665a497b871b42ac4c2778f1 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:04:05.617307 32122 raft_consensus.cc:374] T 3200e86d665a497b871b42ac4c2778f1 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:05.617432 32122 raft_consensus.cc:729] T 3200e86d665a497b871b42ac4c2778f1 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8883498098f4454bbfe563fb83d31899, State: Initialized, Role: FOLLOWER
I20240501 14:04:05.617327 32123 raft_consensus.cc:348] T 0e2a5822d4494ea5b733120bc14c09ea P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:04:05.617528 32123 raft_consensus.cc:374] T 0e2a5822d4494ea5b733120bc14c09ea P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:05.617676 32123 raft_consensus.cc:729] T 0e2a5822d4494ea5b733120bc14c09ea P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0fad54e965647aaaa34b56674646cb4, State: Initialized, Role: FOLLOWER
I20240501 14:04:05.617812 32122 consensus_queue.cc:260] T 3200e86d665a497b871b42ac4c2778f1 P 8883498098f4454bbfe563fb83d31899 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:04:05.618005 32123 consensus_queue.cc:260] T 0e2a5822d4494ea5b733120bc14c09ea P c0fad54e965647aaaa34b56674646cb4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:04:05.619450 32122 ts_tablet_manager.cc:1398] T 3200e86d665a497b871b42ac4c2778f1 P 8883498098f4454bbfe563fb83d31899: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:04:05.620306 32123 ts_tablet_manager.cc:1398] T 0e2a5822d4494ea5b733120bc14c09ea P c0fad54e965647aaaa34b56674646cb4: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:04:05.621968 32125 tablet_bootstrap.cc:492] T abf88dbfc38745a5a5947e425c01dfd8 P 9964c1c76dd14f7dbce766b0c022ab51: No bootstrap required, opened a new log
I20240501 14:04:05.622186 32125 ts_tablet_manager.cc:1367] T abf88dbfc38745a5a5947e425c01dfd8 P 9964c1c76dd14f7dbce766b0c022ab51: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.005s
I20240501 14:04:05.622818 32125 raft_consensus.cc:348] T abf88dbfc38745a5a5947e425c01dfd8 P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:04:05.622987 32125 raft_consensus.cc:374] T abf88dbfc38745a5a5947e425c01dfd8 P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:05.623076 32125 raft_consensus.cc:729] T abf88dbfc38745a5a5947e425c01dfd8 P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9964c1c76dd14f7dbce766b0c022ab51, State: Initialized, Role: FOLLOWER
I20240501 14:04:05.623317 32125 consensus_queue.cc:260] T abf88dbfc38745a5a5947e425c01dfd8 P 9964c1c76dd14f7dbce766b0c022ab51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:04:05.623857 32125 ts_tablet_manager.cc:1398] T abf88dbfc38745a5a5947e425c01dfd8 P 9964c1c76dd14f7dbce766b0c022ab51: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:04:05.647495 32115 raft_consensus.cc:482] T 3200e86d665a497b871b42ac4c2778f1 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:04:05.647665 32115 raft_consensus.cc:504] T 3200e86d665a497b871b42ac4c2778f1 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:04:05.648296 32115 leader_election.cc:288] T 3200e86d665a497b871b42ac4c2778f1 P 870fbdcfdac84bacae0ad22dba25f626 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9964c1c76dd14f7dbce766b0c022ab51 (127.29.124.134:38101), 8883498098f4454bbfe563fb83d31899 (127.29.124.130:43099)
I20240501 14:04:05.648819 31032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3200e86d665a497b871b42ac4c2778f1" candidate_uuid: "870fbdcfdac84bacae0ad22dba25f626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9964c1c76dd14f7dbce766b0c022ab51" is_pre_election: true
I20240501 14:04:05.648907 30508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3200e86d665a497b871b42ac4c2778f1" candidate_uuid: "870fbdcfdac84bacae0ad22dba25f626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8883498098f4454bbfe563fb83d31899" is_pre_election: true
I20240501 14:04:05.649040 31032 raft_consensus.cc:2454] T 3200e86d665a497b871b42ac4c2778f1 P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 870fbdcfdac84bacae0ad22dba25f626 in term 0.
I20240501 14:04:05.649169 30508 raft_consensus.cc:2454] T 3200e86d665a497b871b42ac4c2778f1 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 870fbdcfdac84bacae0ad22dba25f626 in term 0.
I20240501 14:04:05.649451 30704 leader_election.cc:302] T 3200e86d665a497b871b42ac4c2778f1 P 870fbdcfdac84bacae0ad22dba25f626 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 870fbdcfdac84bacae0ad22dba25f626, 9964c1c76dd14f7dbce766b0c022ab51; no voters:
I20240501 14:04:05.649740 32115 raft_consensus.cc:2781] T 3200e86d665a497b871b42ac4c2778f1 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:04:05.649860 32115 raft_consensus.cc:482] T 3200e86d665a497b871b42ac4c2778f1 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:04:05.649946 32115 raft_consensus.cc:3037] T 3200e86d665a497b871b42ac4c2778f1 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:05.651979 32115 raft_consensus.cc:504] T 3200e86d665a497b871b42ac4c2778f1 P 870fbdcfdac84bacae0ad22dba25f626 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:04:05.652776 32115 leader_election.cc:288] T 3200e86d665a497b871b42ac4c2778f1 P 870fbdcfdac84bacae0ad22dba25f626 [CANDIDATE]: Term 1 election: Requested vote from peers 9964c1c76dd14f7dbce766b0c022ab51 (127.29.124.134:38101), 8883498098f4454bbfe563fb83d31899 (127.29.124.130:43099)
I20240501 14:04:05.652902 31032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3200e86d665a497b871b42ac4c2778f1" candidate_uuid: "870fbdcfdac84bacae0ad22dba25f626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9964c1c76dd14f7dbce766b0c022ab51"
I20240501 14:04:05.653014 30508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3200e86d665a497b871b42ac4c2778f1" candidate_uuid: "870fbdcfdac84bacae0ad22dba25f626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8883498098f4454bbfe563fb83d31899"
I20240501 14:04:05.653128 31032 raft_consensus.cc:3037] T 3200e86d665a497b871b42ac4c2778f1 P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:05.653237 30508 raft_consensus.cc:3037] T 3200e86d665a497b871b42ac4c2778f1 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:05.655072 31032 raft_consensus.cc:2454] T 3200e86d665a497b871b42ac4c2778f1 P 9964c1c76dd14f7dbce766b0c022ab51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 870fbdcfdac84bacae0ad22dba25f626 in term 1.
I20240501 14:04:05.655504 30704 leader_election.cc:302] T 3200e86d665a497b871b42ac4c2778f1 P 870fbdcfdac84bacae0ad22dba25f626 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 870fbdcfdac84bacae0ad22dba25f626, 9964c1c76dd14f7dbce766b0c022ab51; no voters:
I20240501 14:04:05.655788 32115 raft_consensus.cc:2781] T 3200e86d665a497b871b42ac4c2778f1 P 870fbdcfdac84bacae0ad22dba25f626 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:04:05.655936 32115 raft_consensus.cc:686] T 3200e86d665a497b871b42ac4c2778f1 P 870fbdcfdac84bacae0ad22dba25f626 [term 1 LEADER]: Becoming Leader. State: Replica: 870fbdcfdac84bacae0ad22dba25f626, State: Running, Role: LEADER
I20240501 14:04:05.656229 30508 raft_consensus.cc:2454] T 3200e86d665a497b871b42ac4c2778f1 P 8883498098f4454bbfe563fb83d31899 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 870fbdcfdac84bacae0ad22dba25f626 in term 1.
I20240501 14:04:05.656459 32115 consensus_queue.cc:237] T 3200e86d665a497b871b42ac4c2778f1 P 870fbdcfdac84bacae0ad22dba25f626 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } }
I20240501 14:04:05.659665 30233 catalog_manager.cc:5484] T 3200e86d665a497b871b42ac4c2778f1 P 870fbdcfdac84bacae0ad22dba25f626 reported cstate change: term changed from 0 to 1, leader changed from <none> to 870fbdcfdac84bacae0ad22dba25f626 (127.29.124.132). New cstate: current_term: 1 leader_uuid: "870fbdcfdac84bacae0ad22dba25f626" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } health_report { overall_health: UNKNOWN } } }
I20240501 14:04:05.764406 31896 raft_consensus.cc:482] T 0e2a5822d4494ea5b733120bc14c09ea P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:04:05.764637 31896 raft_consensus.cc:504] T 0e2a5822d4494ea5b733120bc14c09ea P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:04:05.765334 31896 leader_election.cc:288] T 0e2a5822d4494ea5b733120bc14c09ea P 99f6ca2d65d94b33b7a1d2341c24179a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0fad54e965647aaaa34b56674646cb4 (127.29.124.129:42363), a3e62ccad691413ea2554182f73418a3 (127.29.124.131:43965)
I20240501 14:04:05.765801 30378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e2a5822d4494ea5b733120bc14c09ea" candidate_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0fad54e965647aaaa34b56674646cb4" is_pre_election: true
I20240501 14:04:05.765910 30638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e2a5822d4494ea5b733120bc14c09ea" candidate_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3e62ccad691413ea2554182f73418a3" is_pre_election: true
I20240501 14:04:05.766036 30378 raft_consensus.cc:2454] T 0e2a5822d4494ea5b733120bc14c09ea P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99f6ca2d65d94b33b7a1d2341c24179a in term 0.
I20240501 14:04:05.766186 30638 raft_consensus.cc:2454] T 0e2a5822d4494ea5b733120bc14c09ea P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99f6ca2d65d94b33b7a1d2341c24179a in term 0.
I20240501 14:04:05.766489 30836 leader_election.cc:302] T 0e2a5822d4494ea5b733120bc14c09ea P 99f6ca2d65d94b33b7a1d2341c24179a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99f6ca2d65d94b33b7a1d2341c24179a, c0fad54e965647aaaa34b56674646cb4; no voters:
I20240501 14:04:05.766784 31896 raft_consensus.cc:2781] T 0e2a5822d4494ea5b733120bc14c09ea P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:04:05.766880 31896 raft_consensus.cc:482] T 0e2a5822d4494ea5b733120bc14c09ea P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:04:05.766981 31896 raft_consensus.cc:3037] T 0e2a5822d4494ea5b733120bc14c09ea P 99f6ca2d65d94b33b7a1d2341c24179a [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:05.768733 31896 raft_consensus.cc:504] T 0e2a5822d4494ea5b733120bc14c09ea P 99f6ca2d65d94b33b7a1d2341c24179a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:04:05.769326 31896 leader_election.cc:288] T 0e2a5822d4494ea5b733120bc14c09ea P 99f6ca2d65d94b33b7a1d2341c24179a [CANDIDATE]: Term 1 election: Requested vote from peers c0fad54e965647aaaa34b56674646cb4 (127.29.124.129:42363), a3e62ccad691413ea2554182f73418a3 (127.29.124.131:43965)
I20240501 14:04:05.769668 30378 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e2a5822d4494ea5b733120bc14c09ea" candidate_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0fad54e965647aaaa34b56674646cb4"
I20240501 14:04:05.769812 30638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0e2a5822d4494ea5b733120bc14c09ea" candidate_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a3e62ccad691413ea2554182f73418a3"
I20240501 14:04:05.769912 30378 raft_consensus.cc:3037] T 0e2a5822d4494ea5b733120bc14c09ea P c0fad54e965647aaaa34b56674646cb4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:05.770010 30638 raft_consensus.cc:3037] T 0e2a5822d4494ea5b733120bc14c09ea P a3e62ccad691413ea2554182f73418a3 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:05.772648 30378 raft_consensus.cc:2454] T 0e2a5822d4494ea5b733120bc14c09ea P c0fad54e965647aaaa34b56674646cb4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99f6ca2d65d94b33b7a1d2341c24179a in term 1.
I20240501 14:04:05.772656 30638 raft_consensus.cc:2454] T 0e2a5822d4494ea5b733120bc14c09ea P a3e62ccad691413ea2554182f73418a3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99f6ca2d65d94b33b7a1d2341c24179a in term 1.
I20240501 14:04:05.773062 30836 leader_election.cc:302] T 0e2a5822d4494ea5b733120bc14c09ea P 99f6ca2d65d94b33b7a1d2341c24179a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99f6ca2d65d94b33b7a1d2341c24179a, c0fad54e965647aaaa34b56674646cb4; no voters:
I20240501 14:04:05.773355 31896 raft_consensus.cc:2781] T 0e2a5822d4494ea5b733120bc14c09ea P 99f6ca2d65d94b33b7a1d2341c24179a [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:04:05.773533 31896 raft_consensus.cc:686] T 0e2a5822d4494ea5b733120bc14c09ea P 99f6ca2d65d94b33b7a1d2341c24179a [term 1 LEADER]: Becoming Leader. State: Replica: 99f6ca2d65d94b33b7a1d2341c24179a, State: Running, Role: LEADER
I20240501 14:04:05.773787 31896 consensus_queue.cc:237] T 0e2a5822d4494ea5b733120bc14c09ea P 99f6ca2d65d94b33b7a1d2341c24179a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } }
I20240501 14:04:05.777295 30233 catalog_manager.cc:5484] T 0e2a5822d4494ea5b733120bc14c09ea P 99f6ca2d65d94b33b7a1d2341c24179a reported cstate change: term changed from 0 to 1, leader changed from <none> to 99f6ca2d65d94b33b7a1d2341c24179a (127.29.124.133). New cstate: current_term: 1 leader_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "99f6ca2d65d94b33b7a1d2341c24179a" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35195 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 } health_report { overall_health: UNKNOWN } } }
I20240501 14:04:06.014938 32115 raft_consensus.cc:482] T abf88dbfc38745a5a5947e425c01dfd8 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:04:06.015115 32115 raft_consensus.cc:504] T abf88dbfc38745a5a5947e425c01dfd8 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:04:06.015661 32115 leader_election.cc:288] T abf88dbfc38745a5a5947e425c01dfd8 P 870fbdcfdac84bacae0ad22dba25f626 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9964c1c76dd14f7dbce766b0c022ab51 (127.29.124.134:38101), 8883498098f4454bbfe563fb83d31899 (127.29.124.130:43099)
I20240501 14:04:06.016119 31032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abf88dbfc38745a5a5947e425c01dfd8" candidate_uuid: "870fbdcfdac84bacae0ad22dba25f626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9964c1c76dd14f7dbce766b0c022ab51" is_pre_election: true
I20240501 14:04:06.016176 30508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abf88dbfc38745a5a5947e425c01dfd8" candidate_uuid: "870fbdcfdac84bacae0ad22dba25f626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8883498098f4454bbfe563fb83d31899" is_pre_election: true
I20240501 14:04:06.016372 30508 raft_consensus.cc:2454] T abf88dbfc38745a5a5947e425c01dfd8 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 870fbdcfdac84bacae0ad22dba25f626 in term 0.
I20240501 14:04:06.016372 31032 raft_consensus.cc:2454] T abf88dbfc38745a5a5947e425c01dfd8 P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 870fbdcfdac84bacae0ad22dba25f626 in term 0.
I20240501 14:04:06.016774 30707 leader_election.cc:302] T abf88dbfc38745a5a5947e425c01dfd8 P 870fbdcfdac84bacae0ad22dba25f626 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 870fbdcfdac84bacae0ad22dba25f626, 8883498098f4454bbfe563fb83d31899; no voters:
I20240501 14:04:06.017058 32115 raft_consensus.cc:2781] T abf88dbfc38745a5a5947e425c01dfd8 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:04:06.017155 32115 raft_consensus.cc:482] T abf88dbfc38745a5a5947e425c01dfd8 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:04:06.017305 32115 raft_consensus.cc:3037] T abf88dbfc38745a5a5947e425c01dfd8 P 870fbdcfdac84bacae0ad22dba25f626 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:06.019279 32115 raft_consensus.cc:504] T abf88dbfc38745a5a5947e425c01dfd8 P 870fbdcfdac84bacae0ad22dba25f626 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:04:06.019798 32115 leader_election.cc:288] T abf88dbfc38745a5a5947e425c01dfd8 P 870fbdcfdac84bacae0ad22dba25f626 [CANDIDATE]: Term 1 election: Requested vote from peers 9964c1c76dd14f7dbce766b0c022ab51 (127.29.124.134:38101), 8883498098f4454bbfe563fb83d31899 (127.29.124.130:43099)
I20240501 14:04:06.020109 31032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abf88dbfc38745a5a5947e425c01dfd8" candidate_uuid: "870fbdcfdac84bacae0ad22dba25f626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9964c1c76dd14f7dbce766b0c022ab51"
I20240501 14:04:06.020325 31032 raft_consensus.cc:3037] T abf88dbfc38745a5a5947e425c01dfd8 P 9964c1c76dd14f7dbce766b0c022ab51 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:06.020275 30508 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "abf88dbfc38745a5a5947e425c01dfd8" candidate_uuid: "870fbdcfdac84bacae0ad22dba25f626" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8883498098f4454bbfe563fb83d31899"
I20240501 14:04:06.020457 30508 raft_consensus.cc:3037] T abf88dbfc38745a5a5947e425c01dfd8 P 8883498098f4454bbfe563fb83d31899 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:06.023020 31032 raft_consensus.cc:2454] T abf88dbfc38745a5a5947e425c01dfd8 P 9964c1c76dd14f7dbce766b0c022ab51 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 870fbdcfdac84bacae0ad22dba25f626 in term 1.
I20240501 14:04:06.023038 30508 raft_consensus.cc:2454] T abf88dbfc38745a5a5947e425c01dfd8 P 8883498098f4454bbfe563fb83d31899 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 870fbdcfdac84bacae0ad22dba25f626 in term 1.
I20240501 14:04:06.023406 30704 leader_election.cc:302] T abf88dbfc38745a5a5947e425c01dfd8 P 870fbdcfdac84bacae0ad22dba25f626 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 870fbdcfdac84bacae0ad22dba25f626, 9964c1c76dd14f7dbce766b0c022ab51; no voters:
I20240501 14:04:06.023674 32115 raft_consensus.cc:2781] T abf88dbfc38745a5a5947e425c01dfd8 P 870fbdcfdac84bacae0ad22dba25f626 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:04:06.023928 32115 raft_consensus.cc:686] T abf88dbfc38745a5a5947e425c01dfd8 P 870fbdcfdac84bacae0ad22dba25f626 [term 1 LEADER]: Becoming Leader. State: Replica: 870fbdcfdac84bacae0ad22dba25f626, State: Running, Role: LEADER
I20240501 14:04:06.024248 32115 consensus_queue.cc:237] T abf88dbfc38745a5a5947e425c01dfd8 P 870fbdcfdac84bacae0ad22dba25f626 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } }
I20240501 14:04:06.027499 30233 catalog_manager.cc:5484] T abf88dbfc38745a5a5947e425c01dfd8 P 870fbdcfdac84bacae0ad22dba25f626 reported cstate change: term changed from 0 to 1, leader changed from <none> to 870fbdcfdac84bacae0ad22dba25f626 (127.29.124.132). New cstate: current_term: 1 leader_uuid: "870fbdcfdac84bacae0ad22dba25f626" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "870fbdcfdac84bacae0ad22dba25f626" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 44267 } health_report { overall_health: HEALTHY } } }
I20240501 14:04:06.048147 32115 consensus_queue.cc:1035] T 3200e86d665a497b871b42ac4c2778f1 P 870fbdcfdac84bacae0ad22dba25f626 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:04:06.052464 32025 consensus_queue.cc:1035] T 3200e86d665a497b871b42ac4c2778f1 P 870fbdcfdac84bacae0ad22dba25f626 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:04:06.171456 31896 consensus_queue.cc:1035] T 0e2a5822d4494ea5b733120bc14c09ea P 99f6ca2d65d94b33b7a1d2341c24179a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a3e62ccad691413ea2554182f73418a3" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 43965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:04:06.181113 31896 consensus_queue.cc:1035] T 0e2a5822d4494ea5b733120bc14c09ea P 99f6ca2d65d94b33b7a1d2341c24179a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0fad54e965647aaaa34b56674646cb4" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 42363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:04:06.439358 32115 consensus_queue.cc:1035] T abf88dbfc38745a5a5947e425c01dfd8 P 870fbdcfdac84bacae0ad22dba25f626 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9964c1c76dd14f7dbce766b0c022ab51" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 38101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:04:06.447713 32143 consensus_queue.cc:1035] T abf88dbfc38745a5a5947e425c01dfd8 P 870fbdcfdac84bacae0ad22dba25f626 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8883498098f4454bbfe563fb83d31899" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 43099 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:04:06.478561 32146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:04:06.478987 32146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:04:06.486938 32146 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_11'
INSERT report
rows total: 2000
time total: 219.001 ms
time per row: 0.1095 ms
I20240501 14:04:07.393276 30750 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:04:07.406157 31013 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:07.408167 30489 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:04:07.425843 30882 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:04:07.466434 30357 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:07.468057 30619 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:07.700453 30882 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:04:07.700855 31013 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:07.717809 30357 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:07.720963 30619 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:07.742784 30489 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 14:04:07.745194 30704 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.124.134:38101, user_credentials={real_user=slave}} blocked reactor thread for 46782.5us
I20240501 14:04:07.769142 30750 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:04:08.004413 30750 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:04:08.007777 31013 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:08.018792 30489 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:04:08.019701 30882 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:04:08.052903 30619 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:08.056706 30357 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:08.272078 30750 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:04:08.285553 31013 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:08.290010 30489 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:04:08.295328 30882 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:04:08.312883 30619 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:08.332572 30357 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:08.578318 30489 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:04:08.581061 30750 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:04:08.587064 30882 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:04:08.594538 30619 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:08.596930 30357 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:08.632637 31013 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:09.014631 30750 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:04:09.017204 30882 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:04:09.021867 30357 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:09.026564 30619 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:09.038480 31013 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:09.074784 30489 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
83cd42ea625644ddab1b8a28f3b2898d | 127.29.124.190:35979 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.29.124.148:42681 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
870fbdcfdac84bacae0ad22dba25f626 | 127.29.124.132:44267 | HEALTHY | /B | 8 | 0
8883498098f4454bbfe563fb83d31899 | 127.29.124.130:43099 | HEALTHY | /A | 9 | 0
9964c1c76dd14f7dbce766b0c022ab51 | 127.29.124.134:38101 | HEALTHY | /C | 1 | 0
99f6ca2d65d94b33b7a1d2341c24179a | 127.29.124.133:35195 | HEALTHY | /C | 8 | 0
a3e62ccad691413ea2554182f73418a3 | 127.29.124.131:43965 | HEALTHY | /B | 5 | 0
c0fad54e965647aaaa34b56674646cb4 | 127.29.124.129:42363 | HEALTHY | /A | 5 | 0
Tablet Server Location Summary
Location | Count
----------+---------
/C | 2
/B | 2
/A | 2
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 6 server(s) checked
local_ip_for_outbound_sockets | 127.29.124.129 | experimental | 127.29.124.129:42363
local_ip_for_outbound_sockets | 127.29.124.130 | experimental | 127.29.124.130:43099
local_ip_for_outbound_sockets | 127.29.124.131 | experimental | 127.29.124.131:43965
local_ip_for_outbound_sockets | 127.29.124.132 | experimental | 127.29.124.132:44267
local_ip_for_outbound_sockets | 127.29.124.133 | experimental | 127.29.124.133:35195
local_ip_for_outbound_sockets | 127.29.124.134 | experimental | 127.29.124.134:38101
never_fsync | true | unsafe,advanced | all 6 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 6 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 6 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 6 server(s) checked
server_dump_info_format | pb | hidden | all 6 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.29.124.129:42363
server_dump_info_path | /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.29.124.130:43099
server_dump_info_path | /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.29.124.131:43965
server_dump_info_path | /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.29.124.132:44267
server_dump_info_path | /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.29.124.133:35195
server_dump_info_path | /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.LocationAwareRebalancingBasicTest.Basic.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.29.124.134:38101
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.29.124.148:42681 | all 6 server(s) checked
time_source | builtin | all 6 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 7 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
-------------------------+----+---------+---------------+---------+------------+------------------+-------------
rebalance_test_table_0 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_10 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_11 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_5 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_6 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_7 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_8 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_9 | 3 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 18
First Quartile | 18
Median | 18
Third Quartile | 18
Maximum | 18
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 6
Tables | 12
Tablets | 36
Replicas | 108
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240501 14:04:09.136600 30194 log_verifier.cc:126] Checking tablet 04e140965a314fbca43595aed44942b4
I20240501 14:04:09.173705 30194 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 04e140965a314fbca43595aed44942b4
I20240501 14:04:09.173918 30194 log_verifier.cc:126] Checking tablet 0e2a5822d4494ea5b733120bc14c09ea
I20240501 14:04:09.210347 30194 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 0e2a5822d4494ea5b733120bc14c09ea
I20240501 14:04:09.210466 30194 log_verifier.cc:126] Checking tablet 1b78541b2c4b47cca4e5f22ec9ffd5cd
I20240501 14:04:09.245357 30194 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 1b78541b2c4b47cca4e5f22ec9ffd5cd
I20240501 14:04:09.245481 30194 log_verifier.cc:126] Checking tablet 1f5ff3b1c34a4de780fe4783b9b0eaf1
I20240501 14:04:09.281002 30194 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 1f5ff3b1c34a4de780fe4783b9b0eaf1
I20240501 14:04:09.281124 30194 log_verifier.cc:126] Checking tablet 3200e86d665a497b871b42ac4c2778f1
I20240501 14:04:09.315184 30194 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 3200e86d665a497b871b42ac4c2778f1
I20240501 14:04:09.315311 30194 log_verifier.cc:126] Checking tablet 3e1b9355a8334dbca7276e013cdfa8ff
I20240501 14:04:09.351389 30194 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 3e1b9355a8334dbca7276e013cdfa8ff
I20240501 14:04:09.351507 30194 log_verifier.cc:126] Checking tablet 3e7dddb0e5694c83acde0ee8063c640c
I20240501 14:04:09.386907 30194 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 3e7dddb0e5694c83acde0ee8063c640c
I20240501 14:04:09.387027 30194 log_verifier.cc:126] Checking tablet 3e8c0ae4ee104fd5baf3cf14688656e1
I20240501 14:04:09.427579 30194 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 3e8c0ae4ee104fd5baf3cf14688656e1
I20240501 14:04:09.427717 30194 log_verifier.cc:126] Checking tablet 47d3e9076bdf4ba394ba574849e78ded
I20240501 14:04:09.464228 30194 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 47d3e9076bdf4ba394ba574849e78ded
I20240501 14:04:09.464342 30194 log_verifier.cc:126] Checking tablet 4b44fc1c4c33499a90f6c446110d9c42
I20240501 14:04:09.498366 30194 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 4b44fc1c4c33499a90f6c446110d9c42
I20240501 14:04:09.498500 30194 log_verifier.cc:126] Checking tablet 51ed183d7ec2459faf1b25232b3e504d
I20240501 14:04:09.535262 30194 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 51ed183d7ec2459faf1b25232b3e504d
I20240501 14:04:09.535375 30194 log_verifier.cc:126] Checking tablet 55f4202b341f42eb8e132766b634263b
I20240501 14:04:09.573099 30194 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 55f4202b341f42eb8e132766b634263b
I20240501 14:04:09.573220 30194 log_verifier.cc:126] Checking tablet 63d9b49f760c401ab6b1d483e272729f
I20240501 14:04:09.609407 30194 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 63d9b49f760c401ab6b1d483e272729f
I20240501 14:04:09.609526 30194 log_verifier.cc:126] Checking tablet 66b6d18ec0cc4d8f8f2025666cff4205
I20240501 14:04:09.647972 30194 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 66b6d18ec0cc4d8f8f2025666cff4205
I20240501 14:04:09.648101 30194 log_verifier.cc:126] Checking tablet 68614ecd77ea4d39a47045bc46d0f1ba
I20240501 14:04:09.689191 30194 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 68614ecd77ea4d39a47045bc46d0f1ba
I20240501 14:04:09.689332 30194 log_verifier.cc:126] Checking tablet 7d59b84cb9ed49a58606d439fbd51f07
I20240501 14:04:09.724157 30194 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 7d59b84cb9ed49a58606d439fbd51f07
I20240501 14:04:09.724282 30194 log_verifier.cc:126] Checking tablet 7ff70a696ab64c66a0829a3b390b8c8d
I20240501 14:04:09.758597 30194 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 7ff70a696ab64c66a0829a3b390b8c8d
I20240501 14:04:09.758715 30194 log_verifier.cc:126] Checking tablet 850c9d8997af4db8b1d28707dbd517cb
I20240501 14:04:09.803150 30194 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 850c9d8997af4db8b1d28707dbd517cb
I20240501 14:04:09.803253 30194 log_verifier.cc:126] Checking tablet 8544c28e5c1448e8a069c1842d6638d9
I20240501 14:04:09.838083 30194 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 8544c28e5c1448e8a069c1842d6638d9
I20240501 14:04:09.838212 30194 log_verifier.cc:126] Checking tablet 88537bae3b3c401086e601638bc42e43
I20240501 14:04:09.873337 30194 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 88537bae3b3c401086e601638bc42e43
I20240501 14:04:09.873445 30194 log_verifier.cc:126] Checking tablet 8db48542aec94b11842b21e51dd330c0
I20240501 14:04:09.909945 30194 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 8db48542aec94b11842b21e51dd330c0
I20240501 14:04:09.910053 30194 log_verifier.cc:126] Checking tablet 9051190ef3684094b383b9535e6aea83
I20240501 14:04:09.954448 30194 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 9051190ef3684094b383b9535e6aea83
I20240501 14:04:09.954551 30194 log_verifier.cc:126] Checking tablet 91e42d11fe064d28b457e1bddff845b2
I20240501 14:04:09.990453 30194 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 91e42d11fe064d28b457e1bddff845b2
I20240501 14:04:09.990553 30194 log_verifier.cc:126] Checking tablet 98d4243714a6471b9636b9bd0aad25f2
I20240501 14:04:10.026397 30194 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 98d4243714a6471b9636b9bd0aad25f2
I20240501 14:04:10.026531 30194 log_verifier.cc:126] Checking tablet a0465b32deb54a5cb18fdd132d33285d
I20240501 14:04:10.061601 30194 log_verifier.cc:177] Verified matching terms for 3 ops in tablet a0465b32deb54a5cb18fdd132d33285d
I20240501 14:04:10.061718 30194 log_verifier.cc:126] Checking tablet aa94dbe4c6bc47618980d9dfa4d34333
I20240501 14:04:10.107100 30194 log_verifier.cc:177] Verified matching terms for 3 ops in tablet aa94dbe4c6bc47618980d9dfa4d34333
I20240501 14:04:10.107218 30194 log_verifier.cc:126] Checking tablet abf88dbfc38745a5a5947e425c01dfd8
I20240501 14:04:10.142429 30194 log_verifier.cc:177] Verified matching terms for 3 ops in tablet abf88dbfc38745a5a5947e425c01dfd8
I20240501 14:04:10.142541 30194 log_verifier.cc:126] Checking tablet c6f8a2b1f923426e989e4d2dbcac9e2d
I20240501 14:04:10.176985 30194 log_verifier.cc:177] Verified matching terms for 3 ops in tablet c6f8a2b1f923426e989e4d2dbcac9e2d
I20240501 14:04:10.177110 30194 log_verifier.cc:126] Checking tablet cd60b6c4ba4b4eb58326755b08efb0e9
I20240501 14:04:10.216382 30194 log_verifier.cc:177] Verified matching terms for 3 ops in tablet cd60b6c4ba4b4eb58326755b08efb0e9
I20240501 14:04:10.216507 30194 log_verifier.cc:126] Checking tablet cfaffe32f00e434faf2580b5fbba1fb5
I20240501 14:04:10.252219 30194 log_verifier.cc:177] Verified matching terms for 3 ops in tablet cfaffe32f00e434faf2580b5fbba1fb5
I20240501 14:04:10.252328 30194 log_verifier.cc:126] Checking tablet d8f5779ed9624273b2203b03213e83aa
I20240501 14:04:10.287109 30194 log_verifier.cc:177] Verified matching terms for 3 ops in tablet d8f5779ed9624273b2203b03213e83aa
I20240501 14:04:10.287236 30194 log_verifier.cc:126] Checking tablet dcd50830d6fd4aa3bdcc89c4e60c13ca
I20240501 14:04:10.323220 30194 log_verifier.cc:177] Verified matching terms for 3 ops in tablet dcd50830d6fd4aa3bdcc89c4e60c13ca
I20240501 14:04:10.323328 30194 log_verifier.cc:126] Checking tablet e445777c9d99413a84d7546320cc621d
I20240501 14:04:10.356683 30194 log_verifier.cc:177] Verified matching terms for 3 ops in tablet e445777c9d99413a84d7546320cc621d
I20240501 14:04:10.356809 30194 log_verifier.cc:126] Checking tablet eada160f92e446468cb1a1c78ef4790a
I20240501 14:04:10.393813 30194 log_verifier.cc:177] Verified matching terms for 3 ops in tablet eada160f92e446468cb1a1c78ef4790a
I20240501 14:04:10.393973 30194 log_verifier.cc:126] Checking tablet fc3d4e63dc414a1cbb65d9e0b831c036
I20240501 14:04:10.430413 30194 log_verifier.cc:177] Verified matching terms for 3 ops in tablet fc3d4e63dc414a1cbb65d9e0b831c036
I20240501 14:04:10.430537 30194 log_verifier.cc:126] Checking tablet fc96027b13fe448894ccd93d6fa8a280
I20240501 14:04:10.467168 30194 log_verifier.cc:177] Verified matching terms for 3 ops in tablet fc96027b13fe448894ccd93d6fa8a280
I20240501 14:04:10.488117 30194 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUv7aek/build/asan/bin/kudu pid 30296
I20240501 14:04:10.495438 30417 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:04:10.629747 30194 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUv7aek/build/asan/bin/kudu with pid 30296
W20240501 14:04:10.649204 30575 connection.cc:535] server connection from 127.29.124.129:43685 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:04:10.649688 30194 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUv7aek/build/asan/bin/kudu pid 30426
W20240501 14:04:10.649772 30574 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.29.124.129:42363 (error 108)
W20240501 14:04:10.651777 30574 consensus_peers.cc:485] T 1f5ff3b1c34a4de780fe4783b9b0eaf1 P a3e62ccad691413ea2554182f73418a3 -> Peer c0fad54e965647aaaa34b56674646cb4 (127.29.124.129:42363): Couldn't send request to peer c0fad54e965647aaaa34b56674646cb4. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.129:42363: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:04:10.654096 30836 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.129:42363: connect: Connection refused (error 111)
W20240501 14:04:10.655637 30836 consensus_peers.cc:485] T 0e2a5822d4494ea5b733120bc14c09ea P 99f6ca2d65d94b33b7a1d2341c24179a -> Peer c0fad54e965647aaaa34b56674646cb4 (127.29.124.129:42363): Couldn't send request to peer c0fad54e965647aaaa34b56674646cb4. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.129:42363: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:04:10.655791 30836 consensus_peers.cc:485] T fc3d4e63dc414a1cbb65d9e0b831c036 P 99f6ca2d65d94b33b7a1d2341c24179a -> Peer c0fad54e965647aaaa34b56674646cb4 (127.29.124.129:42363): Couldn't send request to peer c0fad54e965647aaaa34b56674646cb4. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.129:42363: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:04:10.658118 30836 consensus_peers.cc:485] T 91e42d11fe064d28b457e1bddff845b2 P 99f6ca2d65d94b33b7a1d2341c24179a -> Peer c0fad54e965647aaaa34b56674646cb4 (127.29.124.129:42363): Couldn't send request to peer c0fad54e965647aaaa34b56674646cb4. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.129:42363: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:04:10.658305 30548 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 14:04:10.700843 30836 consensus_peers.cc:485] T 3e1b9355a8334dbca7276e013cdfa8ff P 99f6ca2d65d94b33b7a1d2341c24179a -> Peer c0fad54e965647aaaa34b56674646cb4 (127.29.124.129:42363): Couldn't send request to peer c0fad54e965647aaaa34b56674646cb4. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.129:42363: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:04:10.709910 30574 consensus_peers.cc:485] T 850c9d8997af4db8b1d28707dbd517cb P a3e62ccad691413ea2554182f73418a3 -> Peer c0fad54e965647aaaa34b56674646cb4 (127.29.124.129:42363): Couldn't send request to peer c0fad54e965647aaaa34b56674646cb4. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.129:42363: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:04:10.793987 30194 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUv7aek/build/asan/bin/kudu with pid 30426
W20240501 14:04:10.815320 30969 connection.cc:535] server connection from 127.29.124.130:52015 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:04:10.815385 30707 connection.cc:535] client connection to 127.29.124.130:43099 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:04:10.815507 30194 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUv7aek/build/asan/bin/kudu pid 30557
W20240501 14:04:10.815673 30707 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:04:10.817762 30707 consensus_peers.cc:485] T eada160f92e446468cb1a1c78ef4790a P 870fbdcfdac84bacae0ad22dba25f626 -> Peer 8883498098f4454bbfe563fb83d31899 (127.29.124.130:43099): Couldn't send request to peer 8883498098f4454bbfe563fb83d31899. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.130:43099: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:04:10.817945 30707 consensus_peers.cc:485] T 8544c28e5c1448e8a069c1842d6638d9 P 870fbdcfdac84bacae0ad22dba25f626 -> Peer 8883498098f4454bbfe563fb83d31899 (127.29.124.130:43099): Couldn't send request to peer 8883498098f4454bbfe563fb83d31899. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.130:43099: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:04:10.823441 30680 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 14:04:10.835089 30707 consensus_peers.cc:485] T 3200e86d665a497b871b42ac4c2778f1 P 870fbdcfdac84bacae0ad22dba25f626 -> Peer 8883498098f4454bbfe563fb83d31899 (127.29.124.130:43099): Couldn't send request to peer 8883498098f4454bbfe563fb83d31899. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.130:43099: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:04:10.848418 30836 consensus_peers.cc:485] T 63d9b49f760c401ab6b1d483e272729f P 99f6ca2d65d94b33b7a1d2341c24179a -> Peer c0fad54e965647aaaa34b56674646cb4 (127.29.124.129:42363): Couldn't send request to peer c0fad54e965647aaaa34b56674646cb4. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.129:42363: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:04:10.922427 30836 consensus_peers.cc:485] T dcd50830d6fd4aa3bdcc89c4e60c13ca P 99f6ca2d65d94b33b7a1d2341c24179a -> Peer c0fad54e965647aaaa34b56674646cb4 (127.29.124.129:42363): Couldn't send request to peer c0fad54e965647aaaa34b56674646cb4. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.129:42363: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:04:10.935847 30707 consensus_peers.cc:485] T abf88dbfc38745a5a5947e425c01dfd8 P 870fbdcfdac84bacae0ad22dba25f626 -> Peer 8883498098f4454bbfe563fb83d31899 (127.29.124.130:43099): Couldn't send request to peer 8883498098f4454bbfe563fb83d31899. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.130:43099: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:04:10.957343 30194 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUv7aek/build/asan/bin/kudu with pid 30557
I20240501 14:04:10.977334 30194 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUv7aek/build/asan/bin/kudu pid 30688
I20240501 14:04:10.983721 30811 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 14:04:10.997190 30835 consensus_peers.cc:485] T 3e1b9355a8334dbca7276e013cdfa8ff P 99f6ca2d65d94b33b7a1d2341c24179a -> Peer a3e62ccad691413ea2554182f73418a3 (127.29.124.131:43965): Couldn't send request to peer a3e62ccad691413ea2554182f73418a3. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.131:43965: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:04:11.032536 30835 consensus_peers.cc:485] T 0e2a5822d4494ea5b733120bc14c09ea P 99f6ca2d65d94b33b7a1d2341c24179a -> Peer a3e62ccad691413ea2554182f73418a3 (127.29.124.131:43965): Couldn't send request to peer a3e62ccad691413ea2554182f73418a3. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.131:43965: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:04:11.060781 30836 consensus_peers.cc:485] T 3e7dddb0e5694c83acde0ee8063c640c P 99f6ca2d65d94b33b7a1d2341c24179a -> Peer c0fad54e965647aaaa34b56674646cb4 (127.29.124.129:42363): Couldn't send request to peer c0fad54e965647aaaa34b56674646cb4. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.129:42363: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:04:11.077533 30836 consensus_peers.cc:485] T 55f4202b341f42eb8e132766b634263b P 99f6ca2d65d94b33b7a1d2341c24179a -> Peer c0fad54e965647aaaa34b56674646cb4 (127.29.124.129:42363): Couldn't send request to peer c0fad54e965647aaaa34b56674646cb4. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.129:42363: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:04:11.094316 30969 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.130:43099: connect: Connection refused (error 111)
W20240501 14:04:11.095715 30969 consensus_peers.cc:485] T 4b44fc1c4c33499a90f6c446110d9c42 P 9964c1c76dd14f7dbce766b0c022ab51 -> Peer 8883498098f4454bbfe563fb83d31899 (127.29.124.130:43099): Couldn't send request to peer 8883498098f4454bbfe563fb83d31899. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.130:43099: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:04:11.114955 30835 consensus_peers.cc:485] T 63d9b49f760c401ab6b1d483e272729f P 99f6ca2d65d94b33b7a1d2341c24179a -> Peer a3e62ccad691413ea2554182f73418a3 (127.29.124.131:43965): Couldn't send request to peer a3e62ccad691413ea2554182f73418a3. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.131:43965: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:04:11.124429 30835 consensus_peers.cc:485] T 3e7dddb0e5694c83acde0ee8063c640c P 99f6ca2d65d94b33b7a1d2341c24179a -> Peer a3e62ccad691413ea2554182f73418a3 (127.29.124.131:43965): Couldn't send request to peer a3e62ccad691413ea2554182f73418a3. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.131:43965: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:04:11.125449 30707 consensus_peers.cc:485] T 51ed183d7ec2459faf1b25232b3e504d P 870fbdcfdac84bacae0ad22dba25f626 -> Peer 8883498098f4454bbfe563fb83d31899 (127.29.124.130:43099): Couldn't send request to peer 8883498098f4454bbfe563fb83d31899. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.130:43099: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:04:11.125933 30194 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUv7aek/build/asan/bin/kudu with pid 30688
W20240501 14:04:11.146770 30966 connection.cc:535] server connection from 127.29.124.132:52547 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:04:11.146971 30194 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUv7aek/build/asan/bin/kudu pid 30819
I20240501 14:04:11.152936 30942 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:04:11.284489 30194 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUv7aek/build/asan/bin/kudu with pid 30819
W20240501 14:04:11.304466 30222 connection.cc:535] server connection from 127.29.124.133:38841 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:04:11.304574 30194 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUv7aek/build/asan/bin/kudu pid 30950
I20240501 14:04:11.310642 31072 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:04:11.437633 30194 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUv7aek/build/asan/bin/kudu with pid 30950
I20240501 14:04:11.456871 30194 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUv7aek/build/asan/bin/kudu pid 30206
I20240501 14:04:11.457888 30267 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:04:11.608217 30194 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUv7aek/build/asan/bin/kudu with pid 30206
2024-05-01T14:04:11Z chronyd exiting
[ OK ] LocationAwareRebalancingBasicTest.Basic (27764 ms)
[----------] 1 test from LocationAwareRebalancingBasicTest (27764 ms total)
[----------] 1 test from RebalanceStartCriteriaTest
[ RUN ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/1
I20240501 14:04:11.732859 30194 test_util.cc:268] Using random seed: -167666858
I20240501 14:04:11.734596 30194 ts_itest-base.cc:112] Starting cluster with:
I20240501 14:04:11.734674 30194 ts_itest-base.cc:113] --------------
I20240501 14:04:11.734716 30194 ts_itest-base.cc:114] 5 tablet servers
I20240501 14:04:11.734774 30194 ts_itest-base.cc:115] 3 replicas per TS
I20240501 14:04:11.734818 30194 ts_itest-base.cc:116] --------------
2024-05-01T14:04:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T14:04:11Z Disabled control of system clock
I20240501 14:04:11.767107 30194 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUv7aek/build/asan/bin/kudu
/tmp/dist-test-taskUv7aek/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.124.190:39091
--webserver_interface=127.29.124.190
--webserver_port=0
--builtin_ntp_servers=127.29.124.148:36143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.124.190:39091
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240501 14:04:12.090181 32524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:04:12.090497 32524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:04:12.090587 32524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:04:12.098495 32524 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:04:12.098624 32524 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 14:04:12.098677 32524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:04:12.098734 32524 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 14:04:12.098779 32524 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 14:04:12.108042 32524 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.124.148:36143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.29.124.190:39091
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.124.190:39091
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.29.124.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:04:12.109342 32524 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:04:12.111023 32524 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:04:12.122097 32532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:04:12.122138 32529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:04:12.122607 32530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:04:12.122972 32524 server_base.cc:1034] running on GCE node
I20240501 14:04:12.123733 32524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:04:12.125146 32524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:04:12.126351 32524 hybrid_clock.cc:648] HybridClock initialized: now 1714572252126302 us; error 42 us; skew 500 ppm
I20240501 14:04:12.126751 32524 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:04:12.129021 32524 webserver.cc:460] Webserver started at http://127.29.124.190:35005/ using document root <none> and password file <none>
I20240501 14:04:12.129781 32524 fs_manager.cc:359] Metadata directory not provided
I20240501 14:04:12.129887 32524 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:04:12.130208 32524 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:04:12.132766 32524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "ba665546a59f43d8969f9a4df7f789cb"
format_stamp: "Formatted at 2024-05-01 14:04:12 on dist-test-slave-p83k"
I20240501 14:04:12.133637 32524 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "ba665546a59f43d8969f9a4df7f789cb"
format_stamp: "Formatted at 2024-05-01 14:04:12 on dist-test-slave-p83k"
I20240501 14:04:12.138819 32524 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240501 14:04:12.142352 32538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:04:12.143745 32524 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20240501 14:04:12.143920 32524 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/wal
uuid: "ba665546a59f43d8969f9a4df7f789cb"
format_stamp: "Formatted at 2024-05-01 14:04:12 on dist-test-slave-p83k"
I20240501 14:04:12.144194 32524 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:04:12.171098 32524 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:04:12.172170 32524 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:04:12.172453 32524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:04:12.189044 32589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.124.190:39091 every 8 connection(s)
I20240501 14:04:12.189041 32524 rpc_server.cc:307] RPC server started. Bound to: 127.29.124.190:39091
I20240501 14:04:12.190490 32524 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 14:04:12.194133 32590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:12.198173 30194 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUv7aek/build/asan/bin/kudu as pid 32524
I20240501 14:04:12.198419 30194 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 14:04:12.204267 32590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba665546a59f43d8969f9a4df7f789cb: Bootstrap starting.
I20240501 14:04:12.207201 32590 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ba665546a59f43d8969f9a4df7f789cb: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:12.208297 32590 log.cc:824] T 00000000000000000000000000000000 P ba665546a59f43d8969f9a4df7f789cb: Log is configured to *not* fsync() on all Append() calls
I20240501 14:04:12.211050 32590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ba665546a59f43d8969f9a4df7f789cb: No bootstrap required, opened a new log
I20240501 14:04:12.217238 32590 raft_consensus.cc:348] T 00000000000000000000000000000000 P ba665546a59f43d8969f9a4df7f789cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba665546a59f43d8969f9a4df7f789cb" member_type: VOTER last_known_addr { host: "127.29.124.190" port: 39091 } }
I20240501 14:04:12.217612 32590 raft_consensus.cc:374] T 00000000000000000000000000000000 P ba665546a59f43d8969f9a4df7f789cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:12.217703 32590 raft_consensus.cc:729] T 00000000000000000000000000000000 P ba665546a59f43d8969f9a4df7f789cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba665546a59f43d8969f9a4df7f789cb, State: Initialized, Role: FOLLOWER
I20240501 14:04:12.218295 32590 consensus_queue.cc:260] T 00000000000000000000000000000000 P ba665546a59f43d8969f9a4df7f789cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba665546a59f43d8969f9a4df7f789cb" member_type: VOTER last_known_addr { host: "127.29.124.190" port: 39091 } }
I20240501 14:04:12.218501 32590 raft_consensus.cc:388] T 00000000000000000000000000000000 P ba665546a59f43d8969f9a4df7f789cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:04:12.218624 32590 raft_consensus.cc:482] T 00000000000000000000000000000000 P ba665546a59f43d8969f9a4df7f789cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:04:12.218787 32590 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ba665546a59f43d8969f9a4df7f789cb [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:12.220800 32590 raft_consensus.cc:504] T 00000000000000000000000000000000 P ba665546a59f43d8969f9a4df7f789cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba665546a59f43d8969f9a4df7f789cb" member_type: VOTER last_known_addr { host: "127.29.124.190" port: 39091 } }
I20240501 14:04:12.221246 32590 leader_election.cc:302] T 00000000000000000000000000000000 P ba665546a59f43d8969f9a4df7f789cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ba665546a59f43d8969f9a4df7f789cb; no voters:
I20240501 14:04:12.221894 32590 leader_election.cc:288] T 00000000000000000000000000000000 P ba665546a59f43d8969f9a4df7f789cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:04:12.221997 32595 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ba665546a59f43d8969f9a4df7f789cb [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:04:12.222641 32595 raft_consensus.cc:686] T 00000000000000000000000000000000 P ba665546a59f43d8969f9a4df7f789cb [term 1 LEADER]: Becoming Leader. State: Replica: ba665546a59f43d8969f9a4df7f789cb, State: Running, Role: LEADER
I20240501 14:04:12.223163 32595 consensus_queue.cc:237] T 00000000000000000000000000000000 P ba665546a59f43d8969f9a4df7f789cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba665546a59f43d8969f9a4df7f789cb" member_type: VOTER last_known_addr { host: "127.29.124.190" port: 39091 } }
I20240501 14:04:12.223879 32590 sys_catalog.cc:564] T 00000000000000000000000000000000 P ba665546a59f43d8969f9a4df7f789cb [sys.catalog]: configured and running, proceeding with master startup.
I20240501 14:04:12.225759 32596 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba665546a59f43d8969f9a4df7f789cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ba665546a59f43d8969f9a4df7f789cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba665546a59f43d8969f9a4df7f789cb" member_type: VOTER last_known_addr { host: "127.29.124.190" port: 39091 } } }
I20240501 14:04:12.226137 32596 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba665546a59f43d8969f9a4df7f789cb [sys.catalog]: This master's current role is: LEADER
I20240501 14:04:12.226040 32597 sys_catalog.cc:455] T 00000000000000000000000000000000 P ba665546a59f43d8969f9a4df7f789cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader ba665546a59f43d8969f9a4df7f789cb. Latest consensus state: current_term: 1 leader_uuid: "ba665546a59f43d8969f9a4df7f789cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ba665546a59f43d8969f9a4df7f789cb" member_type: VOTER last_known_addr { host: "127.29.124.190" port: 39091 } } }
I20240501 14:04:12.226284 32597 sys_catalog.cc:458] T 00000000000000000000000000000000 P ba665546a59f43d8969f9a4df7f789cb [sys.catalog]: This master's current role is: LEADER
I20240501 14:04:12.228039 32602 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 14:04:12.233156 32602 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 14:04:12.242048 32602 catalog_manager.cc:1347] Generated new cluster ID: e96f14993baf49acba5b7be147ac4dd2
I20240501 14:04:12.242187 32602 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 14:04:12.252000 32602 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 14:04:12.253232 32602 catalog_manager.cc:1504] Loading token signing keys...
I20240501 14:04:12.268383 32602 catalog_manager.cc:5857] T 00000000000000000000000000000000 P ba665546a59f43d8969f9a4df7f789cb: Generated new TSK 0
I20240501 14:04:12.269214 32602 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 14:04:12.281097 30194 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUv7aek/build/asan/bin/kudu
/tmp/dist-test-taskUv7aek/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.124.129:0
--local_ip_for_outbound_sockets=127.29.124.129
--webserver_interface=127.29.124.129
--webserver_port=0
--tserver_master_addrs=127.29.124.190:39091
--builtin_ntp_servers=127.29.124.148:36143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 14:04:12.603103 32614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:04:12.603415 32614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:04:12.603533 32614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:04:12.611021 32614 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:04:12.611179 32614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:04:12.611347 32614 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.124.129
I20240501 14:04:12.620400 32614 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.124.148:36143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.124.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.29.124.129
--webserver_port=0
--tserver_master_addrs=127.29.124.190:39091
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.124.129
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:04:12.621964 32614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:04:12.623764 32614 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:04:12.635855 32619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:04:12.636183 32620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:04:12.636587 32622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:04:12.636973 32614 server_base.cc:1034] running on GCE node
I20240501 14:04:12.637579 32614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:04:12.638499 32614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:04:12.639707 32614 hybrid_clock.cc:648] HybridClock initialized: now 1714572252639657 us; error 39 us; skew 500 ppm
I20240501 14:04:12.640045 32614 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:04:12.642338 32614 webserver.cc:460] Webserver started at http://127.29.124.129:46357/ using document root <none> and password file <none>
I20240501 14:04:12.643082 32614 fs_manager.cc:359] Metadata directory not provided
I20240501 14:04:12.643184 32614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:04:12.643491 32614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:04:12.646211 32614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "e8e904ad3aca4d1fba2ccaa59b708fdf"
format_stamp: "Formatted at 2024-05-01 14:04:12 on dist-test-slave-p83k"
I20240501 14:04:12.646951 32614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "e8e904ad3aca4d1fba2ccaa59b708fdf"
format_stamp: "Formatted at 2024-05-01 14:04:12 on dist-test-slave-p83k"
I20240501 14:04:12.652123 32614 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240501 14:04:12.655520 32628 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:04:12.656864 32614 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 14:04:12.657032 32614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "e8e904ad3aca4d1fba2ccaa59b708fdf"
format_stamp: "Formatted at 2024-05-01 14:04:12 on dist-test-slave-p83k"
I20240501 14:04:12.657220 32614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:04:12.679309 32614 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:04:12.680183 32614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:04:12.680472 32614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:04:12.681455 32614 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:04:12.682933 32614 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:04:12.683036 32614 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:04:12.683147 32614 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:04:12.683219 32614 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:04:12.711232 32614 rpc_server.cc:307] RPC server started. Bound to: 127.29.124.129:41341
I20240501 14:04:12.711282 32740 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.124.129:41341 every 8 connection(s)
I20240501 14:04:12.712509 32614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 14:04:12.722518 30194 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUv7aek/build/asan/bin/kudu as pid 32614
I20240501 14:04:12.722700 30194 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 14:04:12.725196 32741 heartbeater.cc:347] Connected to a master server at 127.29.124.190:39091
I20240501 14:04:12.725535 32741 heartbeater.cc:464] Registering TS with master...
I20240501 14:04:12.726114 30194 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUv7aek/build/asan/bin/kudu
/tmp/dist-test-taskUv7aek/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.124.130:0
--local_ip_for_outbound_sockets=127.29.124.130
--webserver_interface=127.29.124.130
--webserver_port=0
--tserver_master_addrs=127.29.124.190:39091
--builtin_ntp_servers=127.29.124.148:36143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:04:12.726545 32741 heartbeater.cc:511] Master 127.29.124.190:39091 requested a full tablet report, sending...
I20240501 14:04:12.728724 32555 ts_manager.cc:194] Registered new tserver with Master: e8e904ad3aca4d1fba2ccaa59b708fdf (127.29.124.129:41341)
I20240501 14:04:12.730290 32555 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.124.129:51549
W20240501 14:04:13.048624 32745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:04:13.048935 32745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:04:13.049058 32745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:04:13.056417 32745 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:04:13.056563 32745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:04:13.056717 32745 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.124.130
I20240501 14:04:13.065871 32745 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.124.148:36143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.124.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.29.124.130
--webserver_port=0
--tserver_master_addrs=127.29.124.190:39091
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.124.130
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:04:13.067206 32745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:04:13.068961 32745 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:04:13.080361 32750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:04:13.080941 32751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:04:13.081485 32753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:04:13.081640 32745 server_base.cc:1034] running on GCE node
I20240501 14:04:13.082273 32745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:04:13.083205 32745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:04:13.084398 32745 hybrid_clock.cc:648] HybridClock initialized: now 1714572253084346 us; error 52 us; skew 500 ppm
I20240501 14:04:13.084702 32745 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:04:13.086880 32745 webserver.cc:460] Webserver started at http://127.29.124.130:35673/ using document root <none> and password file <none>
I20240501 14:04:13.087608 32745 fs_manager.cc:359] Metadata directory not provided
I20240501 14:04:13.087720 32745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:04:13.088017 32745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:04:13.090673 32745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "a0937eba237a4d14aa25e67c01f4603e"
format_stamp: "Formatted at 2024-05-01 14:04:13 on dist-test-slave-p83k"
I20240501 14:04:13.091413 32745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "a0937eba237a4d14aa25e67c01f4603e"
format_stamp: "Formatted at 2024-05-01 14:04:13 on dist-test-slave-p83k"
I20240501 14:04:13.096550 32745 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240501 14:04:13.099853 32759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:04:13.101191 32745 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 14:04:13.101384 32745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "a0937eba237a4d14aa25e67c01f4603e"
format_stamp: "Formatted at 2024-05-01 14:04:13 on dist-test-slave-p83k"
I20240501 14:04:13.101569 32745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:04:13.136202 32745 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:04:13.137084 32745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:04:13.137387 32745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:04:13.138325 32745 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:04:13.139761 32745 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:04:13.139847 32745 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:04:13.139930 32745 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:04:13.139986 32745 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:04:13.168927 32745 rpc_server.cc:307] RPC server started. Bound to: 127.29.124.130:34227
I20240501 14:04:13.168987 403 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.124.130:34227 every 8 connection(s)
I20240501 14:04:13.170315 32745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 14:04:13.177354 30194 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUv7aek/build/asan/bin/kudu as pid 32745
I20240501 14:04:13.177569 30194 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 14:04:13.180752 30194 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUv7aek/build/asan/bin/kudu
/tmp/dist-test-taskUv7aek/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.124.131:0
--local_ip_for_outbound_sockets=127.29.124.131
--webserver_interface=127.29.124.131
--webserver_port=0
--tserver_master_addrs=127.29.124.190:39091
--builtin_ntp_servers=127.29.124.148:36143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:04:13.187234 404 heartbeater.cc:347] Connected to a master server at 127.29.124.190:39091
I20240501 14:04:13.187619 404 heartbeater.cc:464] Registering TS with master...
I20240501 14:04:13.188658 404 heartbeater.cc:511] Master 127.29.124.190:39091 requested a full tablet report, sending...
I20240501 14:04:13.190423 32555 ts_manager.cc:194] Registered new tserver with Master: a0937eba237a4d14aa25e67c01f4603e (127.29.124.130:34227)
I20240501 14:04:13.191476 32555 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.124.130:35173
W20240501 14:04:13.508733 407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:04:13.509027 407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:04:13.509145 407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:04:13.516680 407 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:04:13.516803 407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:04:13.516958 407 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.124.131
I20240501 14:04:13.526345 407 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.124.148:36143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.124.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.29.124.131
--webserver_port=0
--tserver_master_addrs=127.29.124.190:39091
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.124.131
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:04:13.527725 407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:04:13.529589 407 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:04:13.541731 413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:04:13.541968 415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:04:13.542155 412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:04:13.542559 407 server_base.cc:1034] running on GCE node
I20240501 14:04:13.543169 407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:04:13.544137 407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:04:13.545343 407 hybrid_clock.cc:648] HybridClock initialized: now 1714572253545279 us; error 64 us; skew 500 ppm
I20240501 14:04:13.545665 407 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:04:13.548018 407 webserver.cc:460] Webserver started at http://127.29.124.131:45185/ using document root <none> and password file <none>
I20240501 14:04:13.548772 407 fs_manager.cc:359] Metadata directory not provided
I20240501 14:04:13.548877 407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:04:13.549201 407 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:04:13.552059 407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "b56b6d65e06e4f999e2f05c126390d31"
format_stamp: "Formatted at 2024-05-01 14:04:13 on dist-test-slave-p83k"
I20240501 14:04:13.552819 407 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "b56b6d65e06e4f999e2f05c126390d31"
format_stamp: "Formatted at 2024-05-01 14:04:13 on dist-test-slave-p83k"
I20240501 14:04:13.558077 407 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20240501 14:04:13.561539 421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:04:13.562990 407 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240501 14:04:13.563145 407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "b56b6d65e06e4f999e2f05c126390d31"
format_stamp: "Formatted at 2024-05-01 14:04:13 on dist-test-slave-p83k"
I20240501 14:04:13.563345 407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:04:13.595155 407 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:04:13.596057 407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:04:13.596335 407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:04:13.597330 407 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:04:13.598821 407 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:04:13.598932 407 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:04:13.599040 407 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:04:13.599105 407 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:04:13.628170 407 rpc_server.cc:307] RPC server started. Bound to: 127.29.124.131:45657
I20240501 14:04:13.628240 533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.124.131:45657 every 8 connection(s)
I20240501 14:04:13.629567 407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 14:04:13.633673 30194 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUv7aek/build/asan/bin/kudu as pid 407
I20240501 14:04:13.633849 30194 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 14:04:13.637367 30194 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUv7aek/build/asan/bin/kudu
/tmp/dist-test-taskUv7aek/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.124.132:0
--local_ip_for_outbound_sockets=127.29.124.132
--webserver_interface=127.29.124.132
--webserver_port=0
--tserver_master_addrs=127.29.124.190:39091
--builtin_ntp_servers=127.29.124.148:36143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:04:13.643229 534 heartbeater.cc:347] Connected to a master server at 127.29.124.190:39091
I20240501 14:04:13.643538 534 heartbeater.cc:464] Registering TS with master...
I20240501 14:04:13.644554 534 heartbeater.cc:511] Master 127.29.124.190:39091 requested a full tablet report, sending...
I20240501 14:04:13.646139 32555 ts_manager.cc:194] Registered new tserver with Master: b56b6d65e06e4f999e2f05c126390d31 (127.29.124.131:45657)
I20240501 14:04:13.646905 32555 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.124.131:34981
I20240501 14:04:13.733423 32741 heartbeater.cc:503] Master 127.29.124.190:39091 was elected leader, sending a full tablet report...
W20240501 14:04:13.961767 537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:04:13.962113 537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:04:13.962224 537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:04:13.969878 537 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:04:13.970016 537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:04:13.970130 537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.124.132
I20240501 14:04:13.979305 537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.124.148:36143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.124.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.29.124.132
--webserver_port=0
--tserver_master_addrs=127.29.124.190:39091
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.124.132
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:04:13.980607 537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:04:13.982378 537 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:04:13.993973 543 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:04:13.994365 542 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:04:13.994679 545 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:04:13.994828 537 server_base.cc:1034] running on GCE node
I20240501 14:04:13.995620 537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:04:13.996585 537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:04:13.997783 537 hybrid_clock.cc:648] HybridClock initialized: now 1714572253997733 us; error 47 us; skew 500 ppm
I20240501 14:04:13.998112 537 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:04:14.000321 537 webserver.cc:460] Webserver started at http://127.29.124.132:45643/ using document root <none> and password file <none>
I20240501 14:04:14.001057 537 fs_manager.cc:359] Metadata directory not provided
I20240501 14:04:14.001169 537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:04:14.001521 537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:04:14.004216 537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "e3aa55ee68364d618962a1fcafefccdd"
format_stamp: "Formatted at 2024-05-01 14:04:14 on dist-test-slave-p83k"
I20240501 14:04:14.005216 537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "e3aa55ee68364d618962a1fcafefccdd"
format_stamp: "Formatted at 2024-05-01 14:04:14 on dist-test-slave-p83k"
I20240501 14:04:14.012415 537 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.004s
I20240501 14:04:14.017068 551 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:04:14.018848 537 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20240501 14:04:14.019047 537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "e3aa55ee68364d618962a1fcafefccdd"
format_stamp: "Formatted at 2024-05-01 14:04:14 on dist-test-slave-p83k"
I20240501 14:04:14.019296 537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:04:14.057590 537 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:04:14.058481 537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:04:14.058745 537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:04:14.059779 537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:04:14.061620 537 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:04:14.061707 537 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:04:14.061808 537 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:04:14.061868 537 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:04:14.101429 537 rpc_server.cc:307] RPC server started. Bound to: 127.29.124.132:33391
I20240501 14:04:14.101469 663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.124.132:33391 every 8 connection(s)
I20240501 14:04:14.102777 537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 14:04:14.110743 30194 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUv7aek/build/asan/bin/kudu as pid 537
I20240501 14:04:14.110929 30194 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 14:04:14.114398 30194 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUv7aek/build/asan/bin/kudu
/tmp/dist-test-taskUv7aek/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.124.133:0
--local_ip_for_outbound_sockets=127.29.124.133
--webserver_interface=127.29.124.133
--webserver_port=0
--tserver_master_addrs=127.29.124.190:39091
--builtin_ntp_servers=127.29.124.148:36143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:04:14.117650 664 heartbeater.cc:347] Connected to a master server at 127.29.124.190:39091
I20240501 14:04:14.118000 664 heartbeater.cc:464] Registering TS with master...
I20240501 14:04:14.119068 664 heartbeater.cc:511] Master 127.29.124.190:39091 requested a full tablet report, sending...
I20240501 14:04:14.120934 32555 ts_manager.cc:194] Registered new tserver with Master: e3aa55ee68364d618962a1fcafefccdd (127.29.124.132:33391)
I20240501 14:04:14.122144 32555 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.124.132:52141
I20240501 14:04:14.194571 404 heartbeater.cc:503] Master 127.29.124.190:39091 was elected leader, sending a full tablet report...
W20240501 14:04:14.447261 667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:04:14.447592 667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:04:14.447721 667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:04:14.455361 667 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:04:14.455514 667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:04:14.455665 667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.124.133
I20240501 14:04:14.465023 667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.124.148:36143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.124.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.29.124.133
--webserver_port=0
--tserver_master_addrs=127.29.124.190:39091
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.124.133
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:04:14.466385 667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:04:14.468263 667 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:04:14.480621 672 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:04:14.480785 673 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:04:14.481184 675 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:04:14.481613 667 server_base.cc:1034] running on GCE node
I20240501 14:04:14.482237 667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:04:14.483197 667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:04:14.484418 667 hybrid_clock.cc:648] HybridClock initialized: now 1714572254484363 us; error 50 us; skew 500 ppm
I20240501 14:04:14.484751 667 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:04:14.487031 667 webserver.cc:460] Webserver started at http://127.29.124.133:43573/ using document root <none> and password file <none>
I20240501 14:04:14.487758 667 fs_manager.cc:359] Metadata directory not provided
I20240501 14:04:14.487869 667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:04:14.488188 667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:04:14.490926 667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "a2ae67d159174f04b8b998237ace7136"
format_stamp: "Formatted at 2024-05-01 14:04:14 on dist-test-slave-p83k"
I20240501 14:04:14.491806 667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "a2ae67d159174f04b8b998237ace7136"
format_stamp: "Formatted at 2024-05-01 14:04:14 on dist-test-slave-p83k"
I20240501 14:04:14.497241 667 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240501 14:04:14.500810 681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:04:14.502347 667 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:04:14.502584 667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "a2ae67d159174f04b8b998237ace7136"
format_stamp: "Formatted at 2024-05-01 14:04:14 on dist-test-slave-p83k"
I20240501 14:04:14.502799 667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:04:14.539462 667 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:04:14.540382 667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:04:14.540660 667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:04:14.541705 667 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:04:14.543210 667 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:04:14.543308 667 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:04:14.543423 667 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:04:14.543489 667 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:04:14.572834 667 rpc_server.cc:307] RPC server started. Bound to: 127.29.124.133:45567
I20240501 14:04:14.572903 793 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.124.133:45567 every 8 connection(s)
I20240501 14:04:14.574640 667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 14:04:14.582336 30194 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUv7aek/build/asan/bin/kudu as pid 667
I20240501 14:04:14.582515 30194 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceStartCriteriaTest.UnknownIgnoredTServer_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 14:04:14.588994 794 heartbeater.cc:347] Connected to a master server at 127.29.124.190:39091
I20240501 14:04:14.589315 794 heartbeater.cc:464] Registering TS with master...
I20240501 14:04:14.590207 794 heartbeater.cc:511] Master 127.29.124.190:39091 requested a full tablet report, sending...
I20240501 14:04:14.591522 32555 ts_manager.cc:194] Registered new tserver with Master: a2ae67d159174f04b8b998237ace7136 (127.29.124.133:45567)
I20240501 14:04:14.592298 32555 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.124.133:54365
I20240501 14:04:14.598275 30194 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 14:04:14.614725 32555 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38032:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240501 14:04:14.647468 32676 tablet_service.cc:1467] Processing CreateTablet for tablet 873ec5684b1c430eb9c335a8f6030bcb (DEFAULT_TABLE table=TestTable [id=afae830d20794b6bbcdcb8887622c779]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:14.647799 729 tablet_service.cc:1467] Processing CreateTablet for tablet 873ec5684b1c430eb9c335a8f6030bcb (DEFAULT_TABLE table=TestTable [id=afae830d20794b6bbcdcb8887622c779]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:14.649050 32676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 873ec5684b1c430eb9c335a8f6030bcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:14.649147 729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 873ec5684b1c430eb9c335a8f6030bcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:14.649533 534 heartbeater.cc:503] Master 127.29.124.190:39091 was elected leader, sending a full tablet report...
I20240501 14:04:14.651412 339 tablet_service.cc:1467] Processing CreateTablet for tablet 873ec5684b1c430eb9c335a8f6030bcb (DEFAULT_TABLE table=TestTable [id=afae830d20794b6bbcdcb8887622c779]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:14.652842 339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 873ec5684b1c430eb9c335a8f6030bcb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:14.659934 812 tablet_bootstrap.cc:492] T 873ec5684b1c430eb9c335a8f6030bcb P e8e904ad3aca4d1fba2ccaa59b708fdf: Bootstrap starting.
I20240501 14:04:14.662752 813 tablet_bootstrap.cc:492] T 873ec5684b1c430eb9c335a8f6030bcb P a2ae67d159174f04b8b998237ace7136: Bootstrap starting.
I20240501 14:04:14.665009 812 tablet_bootstrap.cc:654] T 873ec5684b1c430eb9c335a8f6030bcb P e8e904ad3aca4d1fba2ccaa59b708fdf: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:14.665556 814 tablet_bootstrap.cc:492] T 873ec5684b1c430eb9c335a8f6030bcb P a0937eba237a4d14aa25e67c01f4603e: Bootstrap starting.
I20240501 14:04:14.666160 813 tablet_bootstrap.cc:654] T 873ec5684b1c430eb9c335a8f6030bcb P a2ae67d159174f04b8b998237ace7136: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:14.667011 812 log.cc:824] T 873ec5684b1c430eb9c335a8f6030bcb P e8e904ad3aca4d1fba2ccaa59b708fdf: Log is configured to *not* fsync() on all Append() calls
I20240501 14:04:14.667338 813 log.cc:824] T 873ec5684b1c430eb9c335a8f6030bcb P a2ae67d159174f04b8b998237ace7136: Log is configured to *not* fsync() on all Append() calls
I20240501 14:04:14.668788 814 tablet_bootstrap.cc:654] T 873ec5684b1c430eb9c335a8f6030bcb P a0937eba237a4d14aa25e67c01f4603e: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:14.669770 812 tablet_bootstrap.cc:492] T 873ec5684b1c430eb9c335a8f6030bcb P e8e904ad3aca4d1fba2ccaa59b708fdf: No bootstrap required, opened a new log
I20240501 14:04:14.670053 812 ts_tablet_manager.cc:1367] T 873ec5684b1c430eb9c335a8f6030bcb P e8e904ad3aca4d1fba2ccaa59b708fdf: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20240501 14:04:14.670074 814 log.cc:824] T 873ec5684b1c430eb9c335a8f6030bcb P a0937eba237a4d14aa25e67c01f4603e: Log is configured to *not* fsync() on all Append() calls
I20240501 14:04:14.670192 813 tablet_bootstrap.cc:492] T 873ec5684b1c430eb9c335a8f6030bcb P a2ae67d159174f04b8b998237ace7136: No bootstrap required, opened a new log
I20240501 14:04:14.670451 813 ts_tablet_manager.cc:1367] T 873ec5684b1c430eb9c335a8f6030bcb P a2ae67d159174f04b8b998237ace7136: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20240501 14:04:14.673743 814 tablet_bootstrap.cc:492] T 873ec5684b1c430eb9c335a8f6030bcb P a0937eba237a4d14aa25e67c01f4603e: No bootstrap required, opened a new log
I20240501 14:04:14.674027 814 ts_tablet_manager.cc:1367] T 873ec5684b1c430eb9c335a8f6030bcb P a0937eba237a4d14aa25e67c01f4603e: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20240501 14:04:14.676793 812 raft_consensus.cc:348] T 873ec5684b1c430eb9c335a8f6030bcb P e8e904ad3aca4d1fba2ccaa59b708fdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8e904ad3aca4d1fba2ccaa59b708fdf" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 41341 } } peers { permanent_uuid: "a0937eba237a4d14aa25e67c01f4603e" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34227 } } peers { permanent_uuid: "a2ae67d159174f04b8b998237ace7136" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 45567 } }
I20240501 14:04:14.677148 812 raft_consensus.cc:374] T 873ec5684b1c430eb9c335a8f6030bcb P e8e904ad3aca4d1fba2ccaa59b708fdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:14.677279 812 raft_consensus.cc:729] T 873ec5684b1c430eb9c335a8f6030bcb P e8e904ad3aca4d1fba2ccaa59b708fdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8e904ad3aca4d1fba2ccaa59b708fdf, State: Initialized, Role: FOLLOWER
I20240501 14:04:14.678001 812 consensus_queue.cc:260] T 873ec5684b1c430eb9c335a8f6030bcb P e8e904ad3aca4d1fba2ccaa59b708fdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8e904ad3aca4d1fba2ccaa59b708fdf" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 41341 } } peers { permanent_uuid: "a0937eba237a4d14aa25e67c01f4603e" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34227 } } peers { permanent_uuid: "a2ae67d159174f04b8b998237ace7136" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 45567 } }
I20240501 14:04:14.678839 813 raft_consensus.cc:348] T 873ec5684b1c430eb9c335a8f6030bcb P a2ae67d159174f04b8b998237ace7136 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8e904ad3aca4d1fba2ccaa59b708fdf" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 41341 } } peers { permanent_uuid: "a0937eba237a4d14aa25e67c01f4603e" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34227 } } peers { permanent_uuid: "a2ae67d159174f04b8b998237ace7136" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 45567 } }
I20240501 14:04:14.679277 813 raft_consensus.cc:374] T 873ec5684b1c430eb9c335a8f6030bcb P a2ae67d159174f04b8b998237ace7136 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:14.679394 813 raft_consensus.cc:729] T 873ec5684b1c430eb9c335a8f6030bcb P a2ae67d159174f04b8b998237ace7136 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2ae67d159174f04b8b998237ace7136, State: Initialized, Role: FOLLOWER
I20240501 14:04:14.680028 813 consensus_queue.cc:260] T 873ec5684b1c430eb9c335a8f6030bcb P a2ae67d159174f04b8b998237ace7136 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8e904ad3aca4d1fba2ccaa59b708fdf" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 41341 } } peers { permanent_uuid: "a0937eba237a4d14aa25e67c01f4603e" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34227 } } peers { permanent_uuid: "a2ae67d159174f04b8b998237ace7136" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 45567 } }
I20240501 14:04:14.680274 812 ts_tablet_manager.cc:1398] T 873ec5684b1c430eb9c335a8f6030bcb P e8e904ad3aca4d1fba2ccaa59b708fdf: Time spent starting tablet: real 0.010s user 0.006s sys 0.003s
I20240501 14:04:14.681229 794 heartbeater.cc:503] Master 127.29.124.190:39091 was elected leader, sending a full tablet report...
I20240501 14:04:14.681859 813 ts_tablet_manager.cc:1398] T 873ec5684b1c430eb9c335a8f6030bcb P a2ae67d159174f04b8b998237ace7136: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20240501 14:04:14.685694 814 raft_consensus.cc:348] T 873ec5684b1c430eb9c335a8f6030bcb P a0937eba237a4d14aa25e67c01f4603e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8e904ad3aca4d1fba2ccaa59b708fdf" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 41341 } } peers { permanent_uuid: "a0937eba237a4d14aa25e67c01f4603e" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34227 } } peers { permanent_uuid: "a2ae67d159174f04b8b998237ace7136" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 45567 } }
I20240501 14:04:14.686049 814 raft_consensus.cc:374] T 873ec5684b1c430eb9c335a8f6030bcb P a0937eba237a4d14aa25e67c01f4603e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:14.686157 814 raft_consensus.cc:729] T 873ec5684b1c430eb9c335a8f6030bcb P a0937eba237a4d14aa25e67c01f4603e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0937eba237a4d14aa25e67c01f4603e, State: Initialized, Role: FOLLOWER
I20240501 14:04:14.686710 814 consensus_queue.cc:260] T 873ec5684b1c430eb9c335a8f6030bcb P a0937eba237a4d14aa25e67c01f4603e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8e904ad3aca4d1fba2ccaa59b708fdf" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 41341 } } peers { permanent_uuid: "a0937eba237a4d14aa25e67c01f4603e" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34227 } } peers { permanent_uuid: "a2ae67d159174f04b8b998237ace7136" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 45567 } }
I20240501 14:04:14.688023 814 ts_tablet_manager.cc:1398] T 873ec5684b1c430eb9c335a8f6030bcb P a0937eba237a4d14aa25e67c01f4603e: Time spent starting tablet: real 0.014s user 0.009s sys 0.002s
W20240501 14:04:14.715368 32742 tablet.cc:2369] T 873ec5684b1c430eb9c335a8f6030bcb P e8e904ad3aca4d1fba2ccaa59b708fdf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 14:04:14.826813 795 tablet.cc:2369] T 873ec5684b1c430eb9c335a8f6030bcb P a2ae67d159174f04b8b998237ace7136: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:04:14.922626 819 raft_consensus.cc:482] T 873ec5684b1c430eb9c335a8f6030bcb P a2ae67d159174f04b8b998237ace7136 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20240501 14:04:14.923029 405 tablet.cc:2369] T 873ec5684b1c430eb9c335a8f6030bcb P a0937eba237a4d14aa25e67c01f4603e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:04:14.922897 819 raft_consensus.cc:504] T 873ec5684b1c430eb9c335a8f6030bcb P a2ae67d159174f04b8b998237ace7136 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8e904ad3aca4d1fba2ccaa59b708fdf" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 41341 } } peers { permanent_uuid: "a0937eba237a4d14aa25e67c01f4603e" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34227 } } peers { permanent_uuid: "a2ae67d159174f04b8b998237ace7136" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 45567 } }
I20240501 14:04:14.924718 819 leader_election.cc:288] T 873ec5684b1c430eb9c335a8f6030bcb P a2ae67d159174f04b8b998237ace7136 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e8e904ad3aca4d1fba2ccaa59b708fdf (127.29.124.129:41341), a0937eba237a4d14aa25e67c01f4603e (127.29.124.130:34227)
I20240501 14:04:14.931603 32696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "873ec5684b1c430eb9c335a8f6030bcb" candidate_uuid: "a2ae67d159174f04b8b998237ace7136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8e904ad3aca4d1fba2ccaa59b708fdf" is_pre_election: true
I20240501 14:04:14.931892 359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "873ec5684b1c430eb9c335a8f6030bcb" candidate_uuid: "a2ae67d159174f04b8b998237ace7136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0937eba237a4d14aa25e67c01f4603e" is_pre_election: true
I20240501 14:04:14.932057 32696 raft_consensus.cc:2454] T 873ec5684b1c430eb9c335a8f6030bcb P e8e904ad3aca4d1fba2ccaa59b708fdf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2ae67d159174f04b8b998237ace7136 in term 0.
I20240501 14:04:14.932405 359 raft_consensus.cc:2454] T 873ec5684b1c430eb9c335a8f6030bcb P a0937eba237a4d14aa25e67c01f4603e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a2ae67d159174f04b8b998237ace7136 in term 0.
I20240501 14:04:14.932701 685 leader_election.cc:302] T 873ec5684b1c430eb9c335a8f6030bcb P a2ae67d159174f04b8b998237ace7136 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a2ae67d159174f04b8b998237ace7136, e8e904ad3aca4d1fba2ccaa59b708fdf; no voters:
I20240501 14:04:14.933034 819 raft_consensus.cc:2781] T 873ec5684b1c430eb9c335a8f6030bcb P a2ae67d159174f04b8b998237ace7136 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:04:14.933162 819 raft_consensus.cc:482] T 873ec5684b1c430eb9c335a8f6030bcb P a2ae67d159174f04b8b998237ace7136 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:04:14.933296 819 raft_consensus.cc:3037] T 873ec5684b1c430eb9c335a8f6030bcb P a2ae67d159174f04b8b998237ace7136 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:14.935415 819 raft_consensus.cc:504] T 873ec5684b1c430eb9c335a8f6030bcb P a2ae67d159174f04b8b998237ace7136 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8e904ad3aca4d1fba2ccaa59b708fdf" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 41341 } } peers { permanent_uuid: "a0937eba237a4d14aa25e67c01f4603e" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34227 } } peers { permanent_uuid: "a2ae67d159174f04b8b998237ace7136" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 45567 } }
I20240501 14:04:14.935943 819 leader_election.cc:288] T 873ec5684b1c430eb9c335a8f6030bcb P a2ae67d159174f04b8b998237ace7136 [CANDIDATE]: Term 1 election: Requested vote from peers e8e904ad3aca4d1fba2ccaa59b708fdf (127.29.124.129:41341), a0937eba237a4d14aa25e67c01f4603e (127.29.124.130:34227)
I20240501 14:04:14.936391 32696 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "873ec5684b1c430eb9c335a8f6030bcb" candidate_uuid: "a2ae67d159174f04b8b998237ace7136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e8e904ad3aca4d1fba2ccaa59b708fdf"
I20240501 14:04:14.936460 359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "873ec5684b1c430eb9c335a8f6030bcb" candidate_uuid: "a2ae67d159174f04b8b998237ace7136" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a0937eba237a4d14aa25e67c01f4603e"
I20240501 14:04:14.936565 32696 raft_consensus.cc:3037] T 873ec5684b1c430eb9c335a8f6030bcb P e8e904ad3aca4d1fba2ccaa59b708fdf [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:14.936640 359 raft_consensus.cc:3037] T 873ec5684b1c430eb9c335a8f6030bcb P a0937eba237a4d14aa25e67c01f4603e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:14.938673 32696 raft_consensus.cc:2454] T 873ec5684b1c430eb9c335a8f6030bcb P e8e904ad3aca4d1fba2ccaa59b708fdf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2ae67d159174f04b8b998237ace7136 in term 1.
I20240501 14:04:14.938680 359 raft_consensus.cc:2454] T 873ec5684b1c430eb9c335a8f6030bcb P a0937eba237a4d14aa25e67c01f4603e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a2ae67d159174f04b8b998237ace7136 in term 1.
I20240501 14:04:14.939015 685 leader_election.cc:302] T 873ec5684b1c430eb9c335a8f6030bcb P a2ae67d159174f04b8b998237ace7136 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a0937eba237a4d14aa25e67c01f4603e, a2ae67d159174f04b8b998237ace7136; no voters:
I20240501 14:04:14.939293 819 raft_consensus.cc:2781] T 873ec5684b1c430eb9c335a8f6030bcb P a2ae67d159174f04b8b998237ace7136 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:04:14.939689 819 raft_consensus.cc:686] T 873ec5684b1c430eb9c335a8f6030bcb P a2ae67d159174f04b8b998237ace7136 [term 1 LEADER]: Becoming Leader. State: Replica: a2ae67d159174f04b8b998237ace7136, State: Running, Role: LEADER
I20240501 14:04:14.940208 819 consensus_queue.cc:237] T 873ec5684b1c430eb9c335a8f6030bcb P a2ae67d159174f04b8b998237ace7136 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8e904ad3aca4d1fba2ccaa59b708fdf" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 41341 } } peers { permanent_uuid: "a0937eba237a4d14aa25e67c01f4603e" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34227 } } peers { permanent_uuid: "a2ae67d159174f04b8b998237ace7136" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 45567 } }
I20240501 14:04:14.944732 32554 catalog_manager.cc:5484] T 873ec5684b1c430eb9c335a8f6030bcb P a2ae67d159174f04b8b998237ace7136 reported cstate change: term changed from 0 to 1, leader changed from <none> to a2ae67d159174f04b8b998237ace7136 (127.29.124.133). New cstate: current_term: 1 leader_uuid: "a2ae67d159174f04b8b998237ace7136" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e8e904ad3aca4d1fba2ccaa59b708fdf" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 41341 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a0937eba237a4d14aa25e67c01f4603e" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34227 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a2ae67d159174f04b8b998237ace7136" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 45567 } health_report { overall_health: HEALTHY } } }
I20240501 14:04:15.010795 30194 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240501 14:04:15.012972 30194 ts_itest-base.cc:233] Waiting for 1 tablets on tserver e8e904ad3aca4d1fba2ccaa59b708fdf to finish bootstrapping
I20240501 14:04:15.021237 30194 ts_itest-base.cc:233] Waiting for 1 tablets on tserver a0937eba237a4d14aa25e67c01f4603e to finish bootstrapping
I20240501 14:04:15.029116 30194 ts_itest-base.cc:233] Waiting for 0 tablets on tserver b56b6d65e06e4f999e2f05c126390d31 to finish bootstrapping
I20240501 14:04:15.037925 30194 ts_itest-base.cc:233] Waiting for 0 tablets on tserver e3aa55ee68364d618962a1fcafefccdd to finish bootstrapping
I20240501 14:04:15.046195 30194 ts_itest-base.cc:233] Waiting for 1 tablets on tserver a2ae67d159174f04b8b998237ace7136 to finish bootstrapping
I20240501 14:04:15.125033 664 heartbeater.cc:503] Master 127.29.124.190:39091 was elected leader, sending a full tablet report...
I20240501 14:04:15.362026 819 consensus_queue.cc:1035] T 873ec5684b1c430eb9c335a8f6030bcb P a2ae67d159174f04b8b998237ace7136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a0937eba237a4d14aa25e67c01f4603e" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34227 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:04:15.370642 819 consensus_queue.cc:1035] T 873ec5684b1c430eb9c335a8f6030bcb P a2ae67d159174f04b8b998237ace7136 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e8e904ad3aca4d1fba2ccaa59b708fdf" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 41341 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:04:15.509101 469 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:04:15.509887 339 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:04:15.515930 729 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:15.522711 599 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:04:15.523190 32676 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:04:15.687266 339 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:04:15.697813 729 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:15.697813 469 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:04:15.702574 599 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:04:15.703738 32676 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:04:16.321173 339 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:04:16.334009 729 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:16.336500 469 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:04:16.341729 599 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:04:16.344318 32676 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:04:16.492483 339 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:04:16.492746 729 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:16.497620 469 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:04:16.503477 32676 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:04:16.517728 599 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:04:16.675598 339 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:04:16.677431 469 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:04:16.681157 729 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:16.690465 599 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:04:16.706172 32676 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:04:16.848642 30194 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUv7aek/build/asan/bin/kudu pid 32614
I20240501 14:04:16.850040 32736 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:04:16.942466 30194 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUv7aek/build/asan/bin/kudu with pid 32614
W20240501 14:04:16.957996 685 connection.cc:535] client connection to 127.29.124.129:41341 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:04:16.958213 30194 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUv7aek/build/asan/bin/kudu pid 32745
W20240501 14:04:16.958333 685 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:04:16.959153 398 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 14:04:16.960284 685 consensus_peers.cc:485] T 873ec5684b1c430eb9c335a8f6030bcb P a2ae67d159174f04b8b998237ace7136 -> Peer e8e904ad3aca4d1fba2ccaa59b708fdf (127.29.124.129:41341): Couldn't send request to peer e8e904ad3aca4d1fba2ccaa59b708fdf. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.129:41341: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:04:17.052628 30194 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUv7aek/build/asan/bin/kudu with pid 32745
I20240501 14:04:17.067205 30194 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUv7aek/build/asan/bin/kudu pid 407
I20240501 14:04:17.068250 529 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:04:17.157960 30194 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUv7aek/build/asan/bin/kudu with pid 407
I20240501 14:04:17.171818 30194 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUv7aek/build/asan/bin/kudu pid 537
I20240501 14:04:17.172842 658 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:04:17.261006 30194 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUv7aek/build/asan/bin/kudu with pid 537
I20240501 14:04:17.274685 30194 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUv7aek/build/asan/bin/kudu pid 667
I20240501 14:04:17.275684 789 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:04:17.384896 30194 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUv7aek/build/asan/bin/kudu with pid 667
I20240501 14:04:17.398974 30194 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUv7aek/build/asan/bin/kudu pid 32524
I20240501 14:04:17.400008 32585 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:04:17.510018 30194 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUv7aek/build/asan/bin/kudu with pid 32524
2024-05-01T14:04:17Z chronyd exiting
[ OK ] RebalanceStartCriteriaTest.UnknownIgnoredTServer/1 (5815 ms)
[----------] 1 test from RebalanceStartCriteriaTest (5815 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/3
I20240501 14:04:17.548487 30194 test_util.cc:268] Using random seed: -161851218
I20240501 14:04:17.550316 30194 ts_itest-base.cc:112] Starting cluster with:
I20240501 14:04:17.550384 30194 ts_itest-base.cc:113] --------------
I20240501 14:04:17.550441 30194 ts_itest-base.cc:114] 7 tablet servers
I20240501 14:04:17.550485 30194 ts_itest-base.cc:115] 2 replicas per TS
I20240501 14:04:17.550556 30194 ts_itest-base.cc:116] --------------
2024-05-01T14:04:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T14:04:17Z Disabled control of system clock
I20240501 14:04:17.582051 30194 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUv7aek/build/asan/bin/kudu
/tmp/dist-test-taskUv7aek/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.124.190:46779
--webserver_interface=127.29.124.190
--webserver_port=0
--builtin_ntp_servers=127.29.124.148:36129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.124.190:46779
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240501 14:04:17.892823 1021 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240501 14:04:17.893172 1021 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:04:17.893297 1021 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:04:17.893400 1021 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:04:17.900637 1021 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:04:17.900751 1021 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 14:04:17.900841 1021 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:04:17.900902 1021 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 14:04:17.900983 1021 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 14:04:17.909782 1021 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.124.148:36129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.29.124.190:46779
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.124.190:46779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.29.124.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:04:17.911010 1021 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:04:17.912685 1021 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:04:17.922372 1026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:04:17.922677 1027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:04:17.922959 1029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:04:17.923195 1021 server_base.cc:1034] running on GCE node
I20240501 14:04:17.923883 1021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:04:17.925155 1021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:04:17.926366 1021 hybrid_clock.cc:648] HybridClock initialized: now 1714572257926312 us; error 34 us; skew 500 ppm
I20240501 14:04:17.926668 1021 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:04:17.928721 1021 webserver.cc:460] Webserver started at http://127.29.124.190:42637/ using document root <none> and password file <none>
I20240501 14:04:17.929425 1021 fs_manager.cc:359] Metadata directory not provided
I20240501 14:04:17.929551 1021 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:04:17.929821 1021 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:04:17.932231 1021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "3e721134e18641eca52cbebd71ec0520"
format_stamp: "Formatted at 2024-05-01 14:04:17 on dist-test-slave-p83k"
I20240501 14:04:17.932914 1021 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "3e721134e18641eca52cbebd71ec0520"
format_stamp: "Formatted at 2024-05-01 14:04:17 on dist-test-slave-p83k"
I20240501 14:04:17.937618 1021 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20240501 14:04:17.940675 1035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:04:17.941891 1021 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 14:04:17.942040 1021 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/wal
uuid: "3e721134e18641eca52cbebd71ec0520"
format_stamp: "Formatted at 2024-05-01 14:04:17 on dist-test-slave-p83k"
I20240501 14:04:17.942209 1021 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:04:17.960824 1021 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:04:17.961584 1021 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:04:17.961850 1021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:04:17.978148 1021 rpc_server.cc:307] RPC server started. Bound to: 127.29.124.190:46779
I20240501 14:04:17.978184 1086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.124.190:46779 every 8 connection(s)
I20240501 14:04:17.979455 1021 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 14:04:17.983043 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:17.989774 30194 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUv7aek/build/asan/bin/kudu as pid 1021
I20240501 14:04:17.989948 30194 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 14:04:17.991889 1087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e721134e18641eca52cbebd71ec0520: Bootstrap starting.
I20240501 14:04:17.994879 1087 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e721134e18641eca52cbebd71ec0520: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:17.996022 1087 log.cc:824] T 00000000000000000000000000000000 P 3e721134e18641eca52cbebd71ec0520: Log is configured to *not* fsync() on all Append() calls
I20240501 14:04:17.998975 1087 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e721134e18641eca52cbebd71ec0520: No bootstrap required, opened a new log
I20240501 14:04:18.005591 1087 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e721134e18641eca52cbebd71ec0520 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e721134e18641eca52cbebd71ec0520" member_type: VOTER last_known_addr { host: "127.29.124.190" port: 46779 } }
I20240501 14:04:18.005890 1087 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e721134e18641eca52cbebd71ec0520 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:18.005961 1087 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e721134e18641eca52cbebd71ec0520 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e721134e18641eca52cbebd71ec0520, State: Initialized, Role: FOLLOWER
I20240501 14:04:18.006455 1087 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e721134e18641eca52cbebd71ec0520 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e721134e18641eca52cbebd71ec0520" member_type: VOTER last_known_addr { host: "127.29.124.190" port: 46779 } }
I20240501 14:04:18.006621 1087 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e721134e18641eca52cbebd71ec0520 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:04:18.006722 1087 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e721134e18641eca52cbebd71ec0520 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:04:18.006847 1087 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e721134e18641eca52cbebd71ec0520 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:18.009004 1087 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e721134e18641eca52cbebd71ec0520 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e721134e18641eca52cbebd71ec0520" member_type: VOTER last_known_addr { host: "127.29.124.190" port: 46779 } }
I20240501 14:04:18.009472 1087 leader_election.cc:302] T 00000000000000000000000000000000 P 3e721134e18641eca52cbebd71ec0520 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e721134e18641eca52cbebd71ec0520; no voters:
I20240501 14:04:18.009972 1087 leader_election.cc:288] T 00000000000000000000000000000000 P 3e721134e18641eca52cbebd71ec0520 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:04:18.010089 1092 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e721134e18641eca52cbebd71ec0520 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:04:18.010617 1092 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e721134e18641eca52cbebd71ec0520 [term 1 LEADER]: Becoming Leader. State: Replica: 3e721134e18641eca52cbebd71ec0520, State: Running, Role: LEADER
I20240501 14:04:18.011137 1092 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e721134e18641eca52cbebd71ec0520 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e721134e18641eca52cbebd71ec0520" member_type: VOTER last_known_addr { host: "127.29.124.190" port: 46779 } }
I20240501 14:04:18.011719 1087 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e721134e18641eca52cbebd71ec0520 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 14:04:18.013677 1094 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e721134e18641eca52cbebd71ec0520 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e721134e18641eca52cbebd71ec0520. Latest consensus state: current_term: 1 leader_uuid: "3e721134e18641eca52cbebd71ec0520" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e721134e18641eca52cbebd71ec0520" member_type: VOTER last_known_addr { host: "127.29.124.190" port: 46779 } } }
I20240501 14:04:18.014039 1094 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e721134e18641eca52cbebd71ec0520 [sys.catalog]: This master's current role is: LEADER
I20240501 14:04:18.014019 1093 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e721134e18641eca52cbebd71ec0520 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e721134e18641eca52cbebd71ec0520" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e721134e18641eca52cbebd71ec0520" member_type: VOTER last_known_addr { host: "127.29.124.190" port: 46779 } } }
I20240501 14:04:18.014856 1100 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 14:04:18.014885 1093 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e721134e18641eca52cbebd71ec0520 [sys.catalog]: This master's current role is: LEADER
I20240501 14:04:18.019428 1100 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 14:04:18.026584 1100 catalog_manager.cc:1347] Generated new cluster ID: 89c357e2e2be403b8b6489a09421743b
I20240501 14:04:18.026680 1100 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 14:04:18.044808 1100 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 14:04:18.046437 1100 catalog_manager.cc:1504] Loading token signing keys...
I20240501 14:04:18.057951 1100 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 3e721134e18641eca52cbebd71ec0520: Generated new TSK 0
I20240501 14:04:18.058856 1100 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 14:04:18.069872 30194 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUv7aek/build/asan/bin/kudu
/tmp/dist-test-taskUv7aek/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.124.129:0
--local_ip_for_outbound_sockets=127.29.124.129
--webserver_interface=127.29.124.129
--webserver_port=0
--tserver_master_addrs=127.29.124.190:46779
--builtin_ntp_servers=127.29.124.148:36129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 14:04:18.383749 1111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:04:18.384058 1111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:04:18.384187 1111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:04:18.391870 1111 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:04:18.392004 1111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:04:18.392158 1111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.124.129
I20240501 14:04:18.400894 1111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.124.148:36129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.124.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.29.124.129
--webserver_port=0
--tserver_master_addrs=127.29.124.190:46779
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.124.129
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:04:18.402186 1111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:04:18.404187 1111 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:04:18.415619 1116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:04:18.416020 1117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:04:18.416338 1119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:04:18.416893 1111 server_base.cc:1034] running on GCE node
I20240501 14:04:18.417518 1111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:04:18.418443 1111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:04:18.419636 1111 hybrid_clock.cc:648] HybridClock initialized: now 1714572258419589 us; error 38 us; skew 500 ppm
I20240501 14:04:18.419925 1111 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:04:18.425339 1111 webserver.cc:460] Webserver started at http://127.29.124.129:38077/ using document root <none> and password file <none>
I20240501 14:04:18.426056 1111 fs_manager.cc:359] Metadata directory not provided
I20240501 14:04:18.426167 1111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:04:18.426460 1111 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:04:18.428988 1111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "5d45680f41c84ac39bd97c6f00ac2b06"
format_stamp: "Formatted at 2024-05-01 14:04:18 on dist-test-slave-p83k"
I20240501 14:04:18.429754 1111 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "5d45680f41c84ac39bd97c6f00ac2b06"
format_stamp: "Formatted at 2024-05-01 14:04:18 on dist-test-slave-p83k"
I20240501 14:04:18.434506 1111 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20240501 14:04:18.437652 1125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:04:18.438870 1111 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20240501 14:04:18.439016 1111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "5d45680f41c84ac39bd97c6f00ac2b06"
format_stamp: "Formatted at 2024-05-01 14:04:18 on dist-test-slave-p83k"
I20240501 14:04:18.439201 1111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:04:18.460290 1111 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:04:18.461095 1111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:04:18.461421 1111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:04:18.462316 1111 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:04:18.463722 1111 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:04:18.463816 1111 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:04:18.463922 1111 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:04:18.463994 1111 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:04:18.492182 1111 rpc_server.cc:307] RPC server started. Bound to: 127.29.124.129:40925
I20240501 14:04:18.492231 1237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.124.129:40925 every 8 connection(s)
I20240501 14:04:18.493534 1111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 14:04:18.501454 30194 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUv7aek/build/asan/bin/kudu as pid 1111
I20240501 14:04:18.501662 30194 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 14:04:18.505213 30194 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUv7aek/build/asan/bin/kudu
/tmp/dist-test-taskUv7aek/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.124.130:0
--local_ip_for_outbound_sockets=127.29.124.130
--webserver_interface=127.29.124.130
--webserver_port=0
--tserver_master_addrs=127.29.124.190:46779
--builtin_ntp_servers=127.29.124.148:36129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:04:18.505949 1238 heartbeater.cc:347] Connected to a master server at 127.29.124.190:46779
I20240501 14:04:18.506242 1238 heartbeater.cc:464] Registering TS with master...
I20240501 14:04:18.506996 1238 heartbeater.cc:511] Master 127.29.124.190:46779 requested a full tablet report, sending...
I20240501 14:04:18.508953 1052 ts_manager.cc:194] Registered new tserver with Master: 5d45680f41c84ac39bd97c6f00ac2b06 (127.29.124.129:40925)
I20240501 14:04:18.510479 1052 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.124.129:60399
W20240501 14:04:18.819619 1241 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:04:18.819893 1241 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:04:18.820020 1241 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:04:18.827320 1241 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:04:18.827438 1241 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:04:18.827586 1241 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.124.130
I20240501 14:04:18.836297 1241 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.124.148:36129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.124.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.29.124.130
--webserver_port=0
--tserver_master_addrs=127.29.124.190:46779
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.124.130
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:04:18.837611 1241 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:04:18.839293 1241 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:04:18.851048 1247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:04:18.851436 1249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:04:18.851781 1246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:04:18.852196 1241 server_base.cc:1034] running on GCE node
I20240501 14:04:18.852749 1241 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:04:18.853683 1241 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:04:18.854884 1241 hybrid_clock.cc:648] HybridClock initialized: now 1714572258854828 us; error 52 us; skew 500 ppm
I20240501 14:04:18.855183 1241 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:04:18.857216 1241 webserver.cc:460] Webserver started at http://127.29.124.130:41893/ using document root <none> and password file <none>
I20240501 14:04:18.857928 1241 fs_manager.cc:359] Metadata directory not provided
I20240501 14:04:18.858040 1241 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:04:18.858347 1241 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:04:18.860831 1241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "05f4ef3d126d43e9be32731e708ff76c"
format_stamp: "Formatted at 2024-05-01 14:04:18 on dist-test-slave-p83k"
I20240501 14:04:18.861590 1241 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "05f4ef3d126d43e9be32731e708ff76c"
format_stamp: "Formatted at 2024-05-01 14:04:18 on dist-test-slave-p83k"
I20240501 14:04:18.866225 1241 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.002s
I20240501 14:04:18.869233 1255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:04:18.870486 1241 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 14:04:18.870626 1241 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "05f4ef3d126d43e9be32731e708ff76c"
format_stamp: "Formatted at 2024-05-01 14:04:18 on dist-test-slave-p83k"
I20240501 14:04:18.870800 1241 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:04:18.894522 1241 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:04:18.895311 1241 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:04:18.895579 1241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:04:18.896453 1241 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:04:18.897840 1241 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:04:18.897933 1241 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:04:18.898043 1241 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:04:18.898113 1241 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:04:18.926266 1241 rpc_server.cc:307] RPC server started. Bound to: 127.29.124.130:41511
I20240501 14:04:18.926307 1367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.124.130:41511 every 8 connection(s)
I20240501 14:04:18.927542 1241 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 14:04:18.936311 30194 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUv7aek/build/asan/bin/kudu as pid 1241
I20240501 14:04:18.936480 30194 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 14:04:18.939635 1368 heartbeater.cc:347] Connected to a master server at 127.29.124.190:46779
I20240501 14:04:18.939709 30194 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUv7aek/build/asan/bin/kudu
/tmp/dist-test-taskUv7aek/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.124.131:0
--local_ip_for_outbound_sockets=127.29.124.131
--webserver_interface=127.29.124.131
--webserver_port=0
--tserver_master_addrs=127.29.124.190:46779
--builtin_ntp_servers=127.29.124.148:36129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:04:18.939930 1368 heartbeater.cc:464] Registering TS with master...
I20240501 14:04:18.940716 1368 heartbeater.cc:511] Master 127.29.124.190:46779 requested a full tablet report, sending...
I20240501 14:04:18.942021 1052 ts_manager.cc:194] Registered new tserver with Master: 05f4ef3d126d43e9be32731e708ff76c (127.29.124.130:41511)
I20240501 14:04:18.942749 1052 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.124.130:57547
W20240501 14:04:19.252580 1371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:04:19.252842 1371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:04:19.252939 1371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:04:19.260026 1371 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:04:19.260135 1371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:04:19.260247 1371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.124.131
I20240501 14:04:19.268687 1371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.124.148:36129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.124.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.29.124.131
--webserver_port=0
--tserver_master_addrs=127.29.124.190:46779
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.124.131
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:04:19.269923 1371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:04:19.271536 1371 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:04:19.282655 1376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:04:19.282974 1379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:04:19.283567 1377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:04:19.284055 1371 server_base.cc:1034] running on GCE node
I20240501 14:04:19.284566 1371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:04:19.285416 1371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:04:19.286598 1371 hybrid_clock.cc:648] HybridClock initialized: now 1714572259286544 us; error 53 us; skew 500 ppm
I20240501 14:04:19.286870 1371 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:04:19.288836 1371 webserver.cc:460] Webserver started at http://127.29.124.131:43663/ using document root <none> and password file <none>
I20240501 14:04:19.289541 1371 fs_manager.cc:359] Metadata directory not provided
I20240501 14:04:19.289647 1371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:04:19.289930 1371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:04:19.292311 1371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "eb004c2ac9d244f89b61448584d4115b"
format_stamp: "Formatted at 2024-05-01 14:04:19 on dist-test-slave-p83k"
I20240501 14:04:19.293000 1371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "eb004c2ac9d244f89b61448584d4115b"
format_stamp: "Formatted at 2024-05-01 14:04:19 on dist-test-slave-p83k"
I20240501 14:04:19.297569 1371 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20240501 14:04:19.300544 1385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:04:19.301767 1371 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20240501 14:04:19.301913 1371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "eb004c2ac9d244f89b61448584d4115b"
format_stamp: "Formatted at 2024-05-01 14:04:19 on dist-test-slave-p83k"
I20240501 14:04:19.302093 1371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:04:19.330193 1371 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:04:19.330968 1371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:04:19.331228 1371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:04:19.332089 1371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:04:19.333490 1371 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:04:19.333585 1371 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:04:19.333686 1371 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:04:19.333755 1371 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:04:19.361785 1371 rpc_server.cc:307] RPC server started. Bound to: 127.29.124.131:40225
I20240501 14:04:19.361850 1497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.124.131:40225 every 8 connection(s)
I20240501 14:04:19.363021 1371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 14:04:19.369617 30194 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUv7aek/build/asan/bin/kudu as pid 1371
I20240501 14:04:19.369817 30194 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 14:04:19.372820 30194 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUv7aek/build/asan/bin/kudu
/tmp/dist-test-taskUv7aek/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.124.132:0
--local_ip_for_outbound_sockets=127.29.124.132
--webserver_interface=127.29.124.132
--webserver_port=0
--tserver_master_addrs=127.29.124.190:46779
--builtin_ntp_servers=127.29.124.148:36129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:04:19.374904 1498 heartbeater.cc:347] Connected to a master server at 127.29.124.190:46779
I20240501 14:04:19.375207 1498 heartbeater.cc:464] Registering TS with master...
I20240501 14:04:19.376127 1498 heartbeater.cc:511] Master 127.29.124.190:46779 requested a full tablet report, sending...
I20240501 14:04:19.377545 1052 ts_manager.cc:194] Registered new tserver with Master: eb004c2ac9d244f89b61448584d4115b (127.29.124.131:40225)
I20240501 14:04:19.378294 1052 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.124.131:49423
I20240501 14:04:19.513456 1238 heartbeater.cc:503] Master 127.29.124.190:46779 was elected leader, sending a full tablet report...
W20240501 14:04:19.687944 1501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:04:19.688210 1501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:04:19.688306 1501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:04:19.695426 1501 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:04:19.695539 1501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:04:19.695654 1501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.124.132
I20240501 14:04:19.704243 1501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.124.148:36129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.124.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.29.124.132
--webserver_port=0
--tserver_master_addrs=127.29.124.190:46779
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.124.132
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:04:19.705497 1501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:04:19.707211 1501 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:04:19.718837 1509 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:04:19.719287 1506 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:04:19.719647 1507 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:04:19.720134 1501 server_base.cc:1034] running on GCE node
I20240501 14:04:19.720682 1501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:04:19.721698 1501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:04:19.722899 1501 hybrid_clock.cc:648] HybridClock initialized: now 1714572259722850 us; error 38 us; skew 500 ppm
I20240501 14:04:19.723217 1501 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:04:19.725361 1501 webserver.cc:460] Webserver started at http://127.29.124.132:42755/ using document root <none> and password file <none>
I20240501 14:04:19.726061 1501 fs_manager.cc:359] Metadata directory not provided
I20240501 14:04:19.726169 1501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:04:19.726473 1501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:04:19.728950 1501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "953c332688d14831b3bc7cfd895e06e9"
format_stamp: "Formatted at 2024-05-01 14:04:19 on dist-test-slave-p83k"
I20240501 14:04:19.729698 1501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "953c332688d14831b3bc7cfd895e06e9"
format_stamp: "Formatted at 2024-05-01 14:04:19 on dist-test-slave-p83k"
I20240501 14:04:19.734601 1501 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240501 14:04:19.737692 1515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:04:19.738956 1501 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240501 14:04:19.739094 1501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "953c332688d14831b3bc7cfd895e06e9"
format_stamp: "Formatted at 2024-05-01 14:04:19 on dist-test-slave-p83k"
I20240501 14:04:19.739269 1501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:04:19.774470 1501 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:04:19.775386 1501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:04:19.775674 1501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:04:19.776639 1501 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:04:19.778137 1501 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:04:19.778232 1501 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:04:19.778347 1501 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:04:19.778416 1501 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:04:19.806900 1501 rpc_server.cc:307] RPC server started. Bound to: 127.29.124.132:34305
I20240501 14:04:19.807035 1627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.124.132:34305 every 8 connection(s)
I20240501 14:04:19.808157 1501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 14:04:19.813810 30194 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUv7aek/build/asan/bin/kudu as pid 1501
I20240501 14:04:19.814004 30194 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 14:04:19.817166 30194 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUv7aek/build/asan/bin/kudu
/tmp/dist-test-taskUv7aek/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.124.133:0
--local_ip_for_outbound_sockets=127.29.124.133
--webserver_interface=127.29.124.133
--webserver_port=0
--tserver_master_addrs=127.29.124.190:46779
--builtin_ntp_servers=127.29.124.148:36129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:04:19.820647 1628 heartbeater.cc:347] Connected to a master server at 127.29.124.190:46779
I20240501 14:04:19.820925 1628 heartbeater.cc:464] Registering TS with master...
I20240501 14:04:19.821761 1628 heartbeater.cc:511] Master 127.29.124.190:46779 requested a full tablet report, sending...
I20240501 14:04:19.823160 1052 ts_manager.cc:194] Registered new tserver with Master: 953c332688d14831b3bc7cfd895e06e9 (127.29.124.132:34305)
I20240501 14:04:19.824112 1052 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.124.132:54535
I20240501 14:04:19.945212 1368 heartbeater.cc:503] Master 127.29.124.190:46779 was elected leader, sending a full tablet report...
W20240501 14:04:20.143559 1631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:04:20.143864 1631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:04:20.143980 1631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:04:20.151454 1631 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:04:20.151571 1631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:04:20.151722 1631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.124.133
I20240501 14:04:20.160497 1631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.124.148:36129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.124.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.29.124.133
--webserver_port=0
--tserver_master_addrs=127.29.124.190:46779
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.124.133
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:04:20.161785 1631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:04:20.163424 1631 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:04:20.174234 1636 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:04:20.175370 1631 server_base.cc:1034] running on GCE node
W20240501 14:04:20.175771 1639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:04:20.175791 1637 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:04:20.176492 1631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:04:20.177397 1631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:04:20.178599 1631 hybrid_clock.cc:648] HybridClock initialized: now 1714572260178552 us; error 41 us; skew 500 ppm
I20240501 14:04:20.178921 1631 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:04:20.181077 1631 webserver.cc:460] Webserver started at http://127.29.124.133:46115/ using document root <none> and password file <none>
I20240501 14:04:20.181795 1631 fs_manager.cc:359] Metadata directory not provided
I20240501 14:04:20.181898 1631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:04:20.182206 1631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:04:20.184685 1631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "4e6923f9753a4450ac4343c34c427d1e"
format_stamp: "Formatted at 2024-05-01 14:04:20 on dist-test-slave-p83k"
I20240501 14:04:20.185405 1631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "4e6923f9753a4450ac4343c34c427d1e"
format_stamp: "Formatted at 2024-05-01 14:04:20 on dist-test-slave-p83k"
I20240501 14:04:20.190084 1631 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20240501 14:04:20.193133 1645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:04:20.194350 1631 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20240501 14:04:20.194486 1631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "4e6923f9753a4450ac4343c34c427d1e"
format_stamp: "Formatted at 2024-05-01 14:04:20 on dist-test-slave-p83k"
I20240501 14:04:20.194658 1631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:04:20.217667 1631 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:04:20.218426 1631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:04:20.218680 1631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:04:20.219620 1631 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:04:20.220948 1631 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:04:20.221037 1631 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:04:20.221143 1631 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:04:20.221208 1631 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:04:20.249152 1631 rpc_server.cc:307] RPC server started. Bound to: 127.29.124.133:34675
I20240501 14:04:20.249194 1757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.124.133:34675 every 8 connection(s)
I20240501 14:04:20.250443 1631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 14:04:20.259847 30194 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUv7aek/build/asan/bin/kudu as pid 1631
I20240501 14:04:20.260087 30194 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 14:04:20.262313 1758 heartbeater.cc:347] Connected to a master server at 127.29.124.190:46779
I20240501 14:04:20.262578 1758 heartbeater.cc:464] Registering TS with master...
I20240501 14:04:20.263031 30194 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUv7aek/build/asan/bin/kudu
/tmp/dist-test-taskUv7aek/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.124.134:0
--local_ip_for_outbound_sockets=127.29.124.134
--webserver_interface=127.29.124.134
--webserver_port=0
--tserver_master_addrs=127.29.124.190:46779
--builtin_ntp_servers=127.29.124.148:36129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:04:20.263360 1758 heartbeater.cc:511] Master 127.29.124.190:46779 requested a full tablet report, sending...
I20240501 14:04:20.264655 1052 ts_manager.cc:194] Registered new tserver with Master: 4e6923f9753a4450ac4343c34c427d1e (127.29.124.133:34675)
I20240501 14:04:20.265625 1052 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.124.133:34031
I20240501 14:04:20.380771 1498 heartbeater.cc:503] Master 127.29.124.190:46779 was elected leader, sending a full tablet report...
W20240501 14:04:20.578305 1761 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:04:20.578603 1761 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:04:20.578730 1761 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:04:20.586122 1761 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:04:20.586254 1761 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:04:20.586400 1761 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.124.134
I20240501 14:04:20.595366 1761 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.124.148:36129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.124.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.29.124.134
--webserver_port=0
--tserver_master_addrs=127.29.124.190:46779
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.124.134
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:04:20.596709 1761 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:04:20.598556 1761 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:04:20.610589 1769 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:04:20.610632 1766 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:04:20.611137 1767 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:04:20.611694 1761 server_base.cc:1034] running on GCE node
I20240501 14:04:20.612285 1761 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:04:20.613240 1761 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:04:20.614465 1761 hybrid_clock.cc:648] HybridClock initialized: now 1714572260614418 us; error 44 us; skew 500 ppm
I20240501 14:04:20.614770 1761 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:04:20.616917 1761 webserver.cc:460] Webserver started at http://127.29.124.134:41379/ using document root <none> and password file <none>
I20240501 14:04:20.617692 1761 fs_manager.cc:359] Metadata directory not provided
I20240501 14:04:20.617800 1761 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:04:20.618103 1761 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:04:20.620607 1761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "32064197afde4b79bf388b080c365d88"
format_stamp: "Formatted at 2024-05-01 14:04:20 on dist-test-slave-p83k"
I20240501 14:04:20.621352 1761 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "32064197afde4b79bf388b080c365d88"
format_stamp: "Formatted at 2024-05-01 14:04:20 on dist-test-slave-p83k"
I20240501 14:04:20.626147 1761 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.002s
I20240501 14:04:20.629209 1775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:04:20.630524 1761 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240501 14:04:20.630663 1761 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "32064197afde4b79bf388b080c365d88"
format_stamp: "Formatted at 2024-05-01 14:04:20 on dist-test-slave-p83k"
I20240501 14:04:20.630837 1761 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:04:20.648061 1761 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:04:20.648871 1761 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:04:20.649116 1761 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:04:20.650055 1761 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:04:20.651423 1761 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:04:20.651508 1761 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:04:20.651588 1761 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:04:20.651639 1761 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:04:20.677950 1761 rpc_server.cc:307] RPC server started. Bound to: 127.29.124.134:37211
I20240501 14:04:20.677995 1887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.124.134:37211 every 8 connection(s)
I20240501 14:04:20.679205 1761 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 14:04:20.683715 30194 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUv7aek/build/asan/bin/kudu as pid 1761
I20240501 14:04:20.683864 30194 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 14:04:20.686741 30194 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUv7aek/build/asan/bin/kudu
/tmp/dist-test-taskUv7aek/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.124.135:0
--local_ip_for_outbound_sockets=127.29.124.135
--webserver_interface=127.29.124.135
--webserver_port=0
--tserver_master_addrs=127.29.124.190:46779
--builtin_ntp_servers=127.29.124.148:36129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:04:20.691506 1888 heartbeater.cc:347] Connected to a master server at 127.29.124.190:46779
I20240501 14:04:20.691787 1888 heartbeater.cc:464] Registering TS with master...
I20240501 14:04:20.692627 1888 heartbeater.cc:511] Master 127.29.124.190:46779 requested a full tablet report, sending...
I20240501 14:04:20.694054 1052 ts_manager.cc:194] Registered new tserver with Master: 32064197afde4b79bf388b080c365d88 (127.29.124.134:37211)
I20240501 14:04:20.694767 1052 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.124.134:46921
I20240501 14:04:20.827041 1628 heartbeater.cc:503] Master 127.29.124.190:46779 was elected leader, sending a full tablet report...
W20240501 14:04:21.009039 1891 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:04:21.009341 1891 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:04:21.009442 1891 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:04:21.016813 1891 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:04:21.016922 1891 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:04:21.017055 1891 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.124.135
I20240501 14:04:21.025723 1891 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.124.148:36129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.124.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.29.124.135
--webserver_port=0
--tserver_master_addrs=127.29.124.190:46779
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.124.135
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:04:21.027022 1891 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:04:21.028661 1891 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:04:21.040292 1896 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:04:21.040726 1897 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:04:21.040812 1899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:04:21.041240 1891 server_base.cc:1034] running on GCE node
I20240501 14:04:21.041797 1891 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:04:21.042644 1891 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:04:21.043844 1891 hybrid_clock.cc:648] HybridClock initialized: now 1714572261043799 us; error 59 us; skew 500 ppm
I20240501 14:04:21.044157 1891 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:04:21.046334 1891 webserver.cc:460] Webserver started at http://127.29.124.135:41465/ using document root <none> and password file <none>
I20240501 14:04:21.047016 1891 fs_manager.cc:359] Metadata directory not provided
I20240501 14:04:21.047132 1891 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:04:21.047425 1891 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:04:21.049916 1891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "752bc4569da647679d5df51641961840"
format_stamp: "Formatted at 2024-05-01 14:04:21 on dist-test-slave-p83k"
I20240501 14:04:21.050616 1891 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "752bc4569da647679d5df51641961840"
format_stamp: "Formatted at 2024-05-01 14:04:21 on dist-test-slave-p83k"
I20240501 14:04:21.055300 1891 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240501 14:04:21.058408 1905 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:04:21.059598 1891 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:04:21.059746 1891 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "752bc4569da647679d5df51641961840"
format_stamp: "Formatted at 2024-05-01 14:04:21 on dist-test-slave-p83k"
I20240501 14:04:21.059928 1891 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:04:21.088953 1891 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:04:21.089794 1891 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:04:21.090060 1891 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:04:21.090960 1891 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:04:21.092386 1891 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:04:21.092481 1891 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:04:21.092586 1891 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:04:21.092653 1891 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:04:21.122035 1891 rpc_server.cc:307] RPC server started. Bound to: 127.29.124.135:42923
I20240501 14:04:21.122103 2017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.124.135:42923 every 8 connection(s)
I20240501 14:04:21.123369 1891 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 14:04:21.128424 30194 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUv7aek/build/asan/bin/kudu as pid 1891
I20240501 14:04:21.128619 30194 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240501 14:04:21.135857 2018 heartbeater.cc:347] Connected to a master server at 127.29.124.190:46779
I20240501 14:04:21.136133 2018 heartbeater.cc:464] Registering TS with master...
I20240501 14:04:21.136886 2018 heartbeater.cc:511] Master 127.29.124.190:46779 requested a full tablet report, sending...
I20240501 14:04:21.138198 1052 ts_manager.cc:194] Registered new tserver with Master: 752bc4569da647679d5df51641961840 (127.29.124.135:42923)
I20240501 14:04:21.138962 1052 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.124.135:53843
I20240501 14:04:21.144505 30194 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240501 14:04:21.159477 1052 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50932:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240501 14:04:21.202901 1953 tablet_service.cc:1467] Processing CreateTablet for tablet 4da90afe36fd411c8bfd661e4881f86c (DEFAULT_TABLE table=TestTable [id=e0d89fa711ce48dd87e813f93cca2bd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:21.202886 1693 tablet_service.cc:1467] Processing CreateTablet for tablet 4da90afe36fd411c8bfd661e4881f86c (DEFAULT_TABLE table=TestTable [id=e0d89fa711ce48dd87e813f93cca2bd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:21.204228 1953 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4da90afe36fd411c8bfd661e4881f86c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:21.204228 1693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4da90afe36fd411c8bfd661e4881f86c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:21.213590 2034 tablet_bootstrap.cc:492] T 4da90afe36fd411c8bfd661e4881f86c P 4e6923f9753a4450ac4343c34c427d1e: Bootstrap starting.
I20240501 14:04:21.213596 2035 tablet_bootstrap.cc:492] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840: Bootstrap starting.
I20240501 14:04:21.216464 2035 tablet_bootstrap.cc:654] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:21.216945 2034 tablet_bootstrap.cc:654] T 4da90afe36fd411c8bfd661e4881f86c P 4e6923f9753a4450ac4343c34c427d1e: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:21.217434 2035 log.cc:824] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840: Log is configured to *not* fsync() on all Append() calls
I20240501 14:04:21.217993 2034 log.cc:824] T 4da90afe36fd411c8bfd661e4881f86c P 4e6923f9753a4450ac4343c34c427d1e: Log is configured to *not* fsync() on all Append() calls
I20240501 14:04:21.220178 2035 tablet_bootstrap.cc:492] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840: No bootstrap required, opened a new log
I20240501 14:04:21.220350 2034 tablet_bootstrap.cc:492] T 4da90afe36fd411c8bfd661e4881f86c P 4e6923f9753a4450ac4343c34c427d1e: No bootstrap required, opened a new log
I20240501 14:04:21.220458 2035 ts_tablet_manager.cc:1367] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20240501 14:04:21.220585 2034 ts_tablet_manager.cc:1367] T 4da90afe36fd411c8bfd661e4881f86c P 4e6923f9753a4450ac4343c34c427d1e: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20240501 14:04:21.228986 2034 raft_consensus.cc:348] T 4da90afe36fd411c8bfd661e4881f86c P 4e6923f9753a4450ac4343c34c427d1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } }
I20240501 14:04:21.228973 2035 raft_consensus.cc:348] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } }
I20240501 14:04:21.229423 2035 raft_consensus.cc:374] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:21.229415 2034 raft_consensus.cc:374] T 4da90afe36fd411c8bfd661e4881f86c P 4e6923f9753a4450ac4343c34c427d1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:21.229524 2034 raft_consensus.cc:729] T 4da90afe36fd411c8bfd661e4881f86c P 4e6923f9753a4450ac4343c34c427d1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e6923f9753a4450ac4343c34c427d1e, State: Initialized, Role: FOLLOWER
I20240501 14:04:21.229525 2035 raft_consensus.cc:729] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 752bc4569da647679d5df51641961840, State: Initialized, Role: FOLLOWER
I20240501 14:04:21.230199 2034 consensus_queue.cc:260] T 4da90afe36fd411c8bfd661e4881f86c P 4e6923f9753a4450ac4343c34c427d1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } }
I20240501 14:04:21.230198 2035 consensus_queue.cc:260] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } }
I20240501 14:04:21.231297 1758 heartbeater.cc:503] Master 127.29.124.190:46779 was elected leader, sending a full tablet report...
I20240501 14:04:21.231297 2018 heartbeater.cc:503] Master 127.29.124.190:46779 was elected leader, sending a full tablet report...
I20240501 14:04:21.231858 2034 ts_tablet_manager.cc:1398] T 4da90afe36fd411c8bfd661e4881f86c P 4e6923f9753a4450ac4343c34c427d1e: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20240501 14:04:21.231858 2035 ts_tablet_manager.cc:1398] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
W20240501 14:04:21.252353 1759 tablet.cc:2369] T 4da90afe36fd411c8bfd661e4881f86c P 4e6923f9753a4450ac4343c34c427d1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 14:04:21.374948 2019 tablet.cc:2369] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:04:21.375696 2039 raft_consensus.cc:482] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:04:21.375941 2039 raft_consensus.cc:504] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } }
I20240501 14:04:21.377456 2039 leader_election.cc:288] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4e6923f9753a4450ac4343c34c427d1e (127.29.124.133:34675)
I20240501 14:04:21.383106 1713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4da90afe36fd411c8bfd661e4881f86c" candidate_uuid: "752bc4569da647679d5df51641961840" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e6923f9753a4450ac4343c34c427d1e" is_pre_election: true
I20240501 14:04:21.383579 1713 raft_consensus.cc:2454] T 4da90afe36fd411c8bfd661e4881f86c P 4e6923f9753a4450ac4343c34c427d1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 752bc4569da647679d5df51641961840 in term 0.
I20240501 14:04:21.384213 1906 leader_election.cc:302] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4e6923f9753a4450ac4343c34c427d1e, 752bc4569da647679d5df51641961840; no voters:
I20240501 14:04:21.384562 2039 raft_consensus.cc:2781] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:04:21.384685 2039 raft_consensus.cc:482] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:04:21.384783 2039 raft_consensus.cc:3037] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:21.386746 2039 raft_consensus.cc:504] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } }
I20240501 14:04:21.387212 2039 leader_election.cc:288] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840 [CANDIDATE]: Term 1 election: Requested vote from peers 4e6923f9753a4450ac4343c34c427d1e (127.29.124.133:34675)
I20240501 14:04:21.387660 1713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4da90afe36fd411c8bfd661e4881f86c" candidate_uuid: "752bc4569da647679d5df51641961840" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4e6923f9753a4450ac4343c34c427d1e"
I20240501 14:04:21.387833 1713 raft_consensus.cc:3037] T 4da90afe36fd411c8bfd661e4881f86c P 4e6923f9753a4450ac4343c34c427d1e [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:21.389873 1713 raft_consensus.cc:2454] T 4da90afe36fd411c8bfd661e4881f86c P 4e6923f9753a4450ac4343c34c427d1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 752bc4569da647679d5df51641961840 in term 1.
I20240501 14:04:21.390239 1906 leader_election.cc:302] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4e6923f9753a4450ac4343c34c427d1e, 752bc4569da647679d5df51641961840; no voters:
I20240501 14:04:21.390511 2039 raft_consensus.cc:2781] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:04:21.390945 2039 raft_consensus.cc:686] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840 [term 1 LEADER]: Becoming Leader. State: Replica: 752bc4569da647679d5df51641961840, State: Running, Role: LEADER
I20240501 14:04:21.391422 2039 consensus_queue.cc:237] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } }
I20240501 14:04:21.395578 1052 catalog_manager.cc:5484] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840 reported cstate change: term changed from 0 to 1, leader changed from <none> to 752bc4569da647679d5df51641961840 (127.29.124.135). New cstate: current_term: 1 leader_uuid: "752bc4569da647679d5df51641961840" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } health_report { overall_health: HEALTHY } } }
I20240501 14:04:21.431102 30194 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240501 14:04:21.433003 30194 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 5d45680f41c84ac39bd97c6f00ac2b06 to finish bootstrapping
I20240501 14:04:21.441712 30194 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 05f4ef3d126d43e9be32731e708ff76c to finish bootstrapping
I20240501 14:04:21.450124 30194 ts_itest-base.cc:233] Waiting for 0 tablets on tserver eb004c2ac9d244f89b61448584d4115b to finish bootstrapping
I20240501 14:04:21.458407 30194 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 953c332688d14831b3bc7cfd895e06e9 to finish bootstrapping
I20240501 14:04:21.466348 30194 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 4e6923f9753a4450ac4343c34c427d1e to finish bootstrapping
I20240501 14:04:21.473012 30194 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 32064197afde4b79bf388b080c365d88 to finish bootstrapping
I20240501 14:04:21.480796 30194 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 752bc4569da647679d5df51641961840 to finish bootstrapping
I20240501 14:04:21.487747 30194 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUv7aek/build/asan/bin/kudu pid 1501
I20240501 14:04:21.488638 1623 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:04:21.571807 30194 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUv7aek/build/asan/bin/kudu with pid 1501
I20240501 14:04:21.584836 30194 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUv7aek/build/asan/bin/kudu pid 1631
I20240501 14:04:21.585952 1753 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:04:21.669517 30194 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUv7aek/build/asan/bin/kudu with pid 1631
I20240501 14:04:21.683097 30194 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUv7aek/build/asan/bin/kudu pid 1761
I20240501 14:04:21.684144 1883 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:04:21.760542 1888 heartbeater.cc:503] Master 127.29.124.190:46779 was elected leader, sending a full tablet report...
I20240501 14:04:21.761193 30194 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUv7aek/build/asan/bin/kudu with pid 1761
W20240501 14:04:21.773816 1038 connection.cc:535] server connection from 127.29.124.134:46921 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:04:21.774045 30194 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUv7aek/build/asan/bin/kudu pid 1891
I20240501 14:04:21.775067 2013 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:04:21.858211 30194 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUv7aek/build/asan/bin/kudu with pid 1891
I20240501 14:04:25.623015 1052 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50932:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:04:25.658301 1303 tablet_service.cc:1467] Processing CreateTablet for tablet 4c22d4541da544f5b812c6e5e8c2ea40 (DEFAULT_TABLE table=rebalance_test_table_0 [id=817d8db2cf8f4cffb29e21f78e86e2d5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:25.658301 1433 tablet_service.cc:1467] Processing CreateTablet for tablet 4c22d4541da544f5b812c6e5e8c2ea40 (DEFAULT_TABLE table=rebalance_test_table_0 [id=817d8db2cf8f4cffb29e21f78e86e2d5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:25.658339 1432 tablet_service.cc:1467] Processing CreateTablet for tablet 0f837cac32ad4aaa970124639138c2db (DEFAULT_TABLE table=rebalance_test_table_0 [id=817d8db2cf8f4cffb29e21f78e86e2d5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:25.658630 1302 tablet_service.cc:1467] Processing CreateTablet for tablet 8979fd4d21814622ba687edb88fa7851 (DEFAULT_TABLE table=rebalance_test_table_0 [id=817d8db2cf8f4cffb29e21f78e86e2d5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:25.659897 1302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8979fd4d21814622ba687edb88fa7851. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:25.659920 1432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f837cac32ad4aaa970124639138c2db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:25.663758 1303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c22d4541da544f5b812c6e5e8c2ea40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:25.663501 1173 tablet_service.cc:1467] Processing CreateTablet for tablet 8979fd4d21814622ba687edb88fa7851 (DEFAULT_TABLE table=rebalance_test_table_0 [id=817d8db2cf8f4cffb29e21f78e86e2d5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:25.664791 1173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8979fd4d21814622ba687edb88fa7851. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:25.663401 1172 tablet_service.cc:1467] Processing CreateTablet for tablet 0f837cac32ad4aaa970124639138c2db (DEFAULT_TABLE table=rebalance_test_table_0 [id=817d8db2cf8f4cffb29e21f78e86e2d5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:25.665832 1172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f837cac32ad4aaa970124639138c2db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:25.668691 1433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c22d4541da544f5b812c6e5e8c2ea40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:25.678844 2062 tablet_bootstrap.cc:492] T 4c22d4541da544f5b812c6e5e8c2ea40 P 05f4ef3d126d43e9be32731e708ff76c: Bootstrap starting.
I20240501 14:04:25.679277 2063 tablet_bootstrap.cc:492] T 0f837cac32ad4aaa970124639138c2db P eb004c2ac9d244f89b61448584d4115b: Bootstrap starting.
I20240501 14:04:25.681367 2062 tablet_bootstrap.cc:654] T 4c22d4541da544f5b812c6e5e8c2ea40 P 05f4ef3d126d43e9be32731e708ff76c: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:25.682274 2062 log.cc:824] T 4c22d4541da544f5b812c6e5e8c2ea40 P 05f4ef3d126d43e9be32731e708ff76c: Log is configured to *not* fsync() on all Append() calls
I20240501 14:04:25.682411 2063 tablet_bootstrap.cc:654] T 0f837cac32ad4aaa970124639138c2db P eb004c2ac9d244f89b61448584d4115b: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:25.683538 2063 log.cc:824] T 0f837cac32ad4aaa970124639138c2db P eb004c2ac9d244f89b61448584d4115b: Log is configured to *not* fsync() on all Append() calls
I20240501 14:04:25.684887 2065 tablet_bootstrap.cc:492] T 0f837cac32ad4aaa970124639138c2db P 5d45680f41c84ac39bd97c6f00ac2b06: Bootstrap starting.
I20240501 14:04:25.685437 2062 tablet_bootstrap.cc:492] T 4c22d4541da544f5b812c6e5e8c2ea40 P 05f4ef3d126d43e9be32731e708ff76c: No bootstrap required, opened a new log
I20240501 14:04:25.685748 2062 ts_tablet_manager.cc:1367] T 4c22d4541da544f5b812c6e5e8c2ea40 P 05f4ef3d126d43e9be32731e708ff76c: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.004s
I20240501 14:04:25.686920 2063 tablet_bootstrap.cc:492] T 0f837cac32ad4aaa970124639138c2db P eb004c2ac9d244f89b61448584d4115b: No bootstrap required, opened a new log
I20240501 14:04:25.687180 2063 ts_tablet_manager.cc:1367] T 0f837cac32ad4aaa970124639138c2db P eb004c2ac9d244f89b61448584d4115b: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20240501 14:04:25.687570 2065 tablet_bootstrap.cc:654] T 0f837cac32ad4aaa970124639138c2db P 5d45680f41c84ac39bd97c6f00ac2b06: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:25.688390 2065 log.cc:824] T 0f837cac32ad4aaa970124639138c2db P 5d45680f41c84ac39bd97c6f00ac2b06: Log is configured to *not* fsync() on all Append() calls
I20240501 14:04:25.690841 2065 tablet_bootstrap.cc:492] T 0f837cac32ad4aaa970124639138c2db P 5d45680f41c84ac39bd97c6f00ac2b06: No bootstrap required, opened a new log
I20240501 14:04:25.691061 2065 ts_tablet_manager.cc:1367] T 0f837cac32ad4aaa970124639138c2db P 5d45680f41c84ac39bd97c6f00ac2b06: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240501 14:04:25.694026 2062 raft_consensus.cc:348] T 4c22d4541da544f5b812c6e5e8c2ea40 P 05f4ef3d126d43e9be32731e708ff76c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } }
I20240501 14:04:25.694423 2062 raft_consensus.cc:374] T 4c22d4541da544f5b812c6e5e8c2ea40 P 05f4ef3d126d43e9be32731e708ff76c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:25.694522 2062 raft_consensus.cc:729] T 4c22d4541da544f5b812c6e5e8c2ea40 P 05f4ef3d126d43e9be32731e708ff76c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4ef3d126d43e9be32731e708ff76c, State: Initialized, Role: FOLLOWER
I20240501 14:04:25.695158 2062 consensus_queue.cc:260] T 4c22d4541da544f5b812c6e5e8c2ea40 P 05f4ef3d126d43e9be32731e708ff76c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } }
I20240501 14:04:25.695374 2063 raft_consensus.cc:348] T 0f837cac32ad4aaa970124639138c2db P eb004c2ac9d244f89b61448584d4115b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } }
I20240501 14:04:25.695788 2063 raft_consensus.cc:374] T 0f837cac32ad4aaa970124639138c2db P eb004c2ac9d244f89b61448584d4115b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:25.695919 2063 raft_consensus.cc:729] T 0f837cac32ad4aaa970124639138c2db P eb004c2ac9d244f89b61448584d4115b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb004c2ac9d244f89b61448584d4115b, State: Initialized, Role: FOLLOWER
I20240501 14:04:25.696542 2063 consensus_queue.cc:260] T 0f837cac32ad4aaa970124639138c2db P eb004c2ac9d244f89b61448584d4115b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } }
I20240501 14:04:25.696759 2062 ts_tablet_manager.cc:1398] T 4c22d4541da544f5b812c6e5e8c2ea40 P 05f4ef3d126d43e9be32731e708ff76c: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20240501 14:04:25.697202 2062 tablet_bootstrap.cc:492] T 8979fd4d21814622ba687edb88fa7851 P 05f4ef3d126d43e9be32731e708ff76c: Bootstrap starting.
I20240501 14:04:25.698065 2063 ts_tablet_manager.cc:1398] T 0f837cac32ad4aaa970124639138c2db P eb004c2ac9d244f89b61448584d4115b: Time spent starting tablet: real 0.011s user 0.012s sys 0.000s
I20240501 14:04:25.698467 2063 tablet_bootstrap.cc:492] T 4c22d4541da544f5b812c6e5e8c2ea40 P eb004c2ac9d244f89b61448584d4115b: Bootstrap starting.
I20240501 14:04:25.698789 2062 tablet_bootstrap.cc:654] T 8979fd4d21814622ba687edb88fa7851 P 05f4ef3d126d43e9be32731e708ff76c: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:25.699919 2063 tablet_bootstrap.cc:654] T 4c22d4541da544f5b812c6e5e8c2ea40 P eb004c2ac9d244f89b61448584d4115b: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:25.702406 2062 tablet_bootstrap.cc:492] T 8979fd4d21814622ba687edb88fa7851 P 05f4ef3d126d43e9be32731e708ff76c: No bootstrap required, opened a new log
I20240501 14:04:25.702617 2062 ts_tablet_manager.cc:1367] T 8979fd4d21814622ba687edb88fa7851 P 05f4ef3d126d43e9be32731e708ff76c: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.001s
I20240501 14:04:25.702549 2065 raft_consensus.cc:348] T 0f837cac32ad4aaa970124639138c2db P 5d45680f41c84ac39bd97c6f00ac2b06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } }
I20240501 14:04:25.702919 2065 raft_consensus.cc:374] T 0f837cac32ad4aaa970124639138c2db P 5d45680f41c84ac39bd97c6f00ac2b06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:25.703015 2065 raft_consensus.cc:729] T 0f837cac32ad4aaa970124639138c2db P 5d45680f41c84ac39bd97c6f00ac2b06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d45680f41c84ac39bd97c6f00ac2b06, State: Initialized, Role: FOLLOWER
I20240501 14:04:25.703518 2062 raft_consensus.cc:348] T 8979fd4d21814622ba687edb88fa7851 P 05f4ef3d126d43e9be32731e708ff76c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } }
I20240501 14:04:25.703565 2065 consensus_queue.cc:260] T 0f837cac32ad4aaa970124639138c2db P 5d45680f41c84ac39bd97c6f00ac2b06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } }
I20240501 14:04:25.703712 2062 raft_consensus.cc:374] T 8979fd4d21814622ba687edb88fa7851 P 05f4ef3d126d43e9be32731e708ff76c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:25.703842 2062 raft_consensus.cc:729] T 8979fd4d21814622ba687edb88fa7851 P 05f4ef3d126d43e9be32731e708ff76c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4ef3d126d43e9be32731e708ff76c, State: Initialized, Role: FOLLOWER
I20240501 14:04:25.704195 2062 consensus_queue.cc:260] T 8979fd4d21814622ba687edb88fa7851 P 05f4ef3d126d43e9be32731e708ff76c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } }
I20240501 14:04:25.704720 2062 ts_tablet_manager.cc:1398] T 8979fd4d21814622ba687edb88fa7851 P 05f4ef3d126d43e9be32731e708ff76c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:04:25.706293 2065 ts_tablet_manager.cc:1398] T 0f837cac32ad4aaa970124639138c2db P 5d45680f41c84ac39bd97c6f00ac2b06: Time spent starting tablet: real 0.015s user 0.010s sys 0.000s
I20240501 14:04:25.706696 2065 tablet_bootstrap.cc:492] T 8979fd4d21814622ba687edb88fa7851 P 5d45680f41c84ac39bd97c6f00ac2b06: Bootstrap starting.
I20240501 14:04:25.708285 2065 tablet_bootstrap.cc:654] T 8979fd4d21814622ba687edb88fa7851 P 5d45680f41c84ac39bd97c6f00ac2b06: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:25.710156 2063 tablet_bootstrap.cc:492] T 4c22d4541da544f5b812c6e5e8c2ea40 P eb004c2ac9d244f89b61448584d4115b: No bootstrap required, opened a new log
I20240501 14:04:25.710291 2063 ts_tablet_manager.cc:1367] T 4c22d4541da544f5b812c6e5e8c2ea40 P eb004c2ac9d244f89b61448584d4115b: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.005s
I20240501 14:04:25.710381 2065 tablet_bootstrap.cc:492] T 8979fd4d21814622ba687edb88fa7851 P 5d45680f41c84ac39bd97c6f00ac2b06: No bootstrap required, opened a new log
I20240501 14:04:25.710505 2065 ts_tablet_manager.cc:1367] T 8979fd4d21814622ba687edb88fa7851 P 5d45680f41c84ac39bd97c6f00ac2b06: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:04:25.710996 2063 raft_consensus.cc:348] T 4c22d4541da544f5b812c6e5e8c2ea40 P eb004c2ac9d244f89b61448584d4115b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } }
I20240501 14:04:25.711145 2063 raft_consensus.cc:374] T 4c22d4541da544f5b812c6e5e8c2ea40 P eb004c2ac9d244f89b61448584d4115b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:25.711134 2065 raft_consensus.cc:348] T 8979fd4d21814622ba687edb88fa7851 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } }
I20240501 14:04:25.711231 2063 raft_consensus.cc:729] T 4c22d4541da544f5b812c6e5e8c2ea40 P eb004c2ac9d244f89b61448584d4115b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb004c2ac9d244f89b61448584d4115b, State: Initialized, Role: FOLLOWER
I20240501 14:04:25.711292 2065 raft_consensus.cc:374] T 8979fd4d21814622ba687edb88fa7851 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:25.711413 2065 raft_consensus.cc:729] T 8979fd4d21814622ba687edb88fa7851 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d45680f41c84ac39bd97c6f00ac2b06, State: Initialized, Role: FOLLOWER
I20240501 14:04:25.711450 2063 consensus_queue.cc:260] T 4c22d4541da544f5b812c6e5e8c2ea40 P eb004c2ac9d244f89b61448584d4115b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } }
I20240501 14:04:25.711624 2065 consensus_queue.cc:260] T 8979fd4d21814622ba687edb88fa7851 P 5d45680f41c84ac39bd97c6f00ac2b06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } }
I20240501 14:04:25.711933 2063 ts_tablet_manager.cc:1398] T 4c22d4541da544f5b812c6e5e8c2ea40 P eb004c2ac9d244f89b61448584d4115b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:04:25.712100 2065 ts_tablet_manager.cc:1398] T 8979fd4d21814622ba687edb88fa7851 P 5d45680f41c84ac39bd97c6f00ac2b06: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20240501 14:04:25.749480 1239 tablet.cc:2369] T 0f837cac32ad4aaa970124639138c2db P 5d45680f41c84ac39bd97c6f00ac2b06: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:04:25.811640 2068 raft_consensus.cc:482] T 4c22d4541da544f5b812c6e5e8c2ea40 P 05f4ef3d126d43e9be32731e708ff76c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:04:25.811933 2068 raft_consensus.cc:504] T 4c22d4541da544f5b812c6e5e8c2ea40 P 05f4ef3d126d43e9be32731e708ff76c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } }
I20240501 14:04:25.813541 2068 leader_election.cc:288] T 4c22d4541da544f5b812c6e5e8c2ea40 P 05f4ef3d126d43e9be32731e708ff76c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb004c2ac9d244f89b61448584d4115b (127.29.124.131:40225)
I20240501 14:04:25.819658 1453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c22d4541da544f5b812c6e5e8c2ea40" candidate_uuid: "05f4ef3d126d43e9be32731e708ff76c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb004c2ac9d244f89b61448584d4115b" is_pre_election: true
I20240501 14:04:25.820106 1453 raft_consensus.cc:2454] T 4c22d4541da544f5b812c6e5e8c2ea40 P eb004c2ac9d244f89b61448584d4115b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05f4ef3d126d43e9be32731e708ff76c in term 0.
I20240501 14:04:25.820694 1259 leader_election.cc:302] T 4c22d4541da544f5b812c6e5e8c2ea40 P 05f4ef3d126d43e9be32731e708ff76c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 05f4ef3d126d43e9be32731e708ff76c, eb004c2ac9d244f89b61448584d4115b; no voters:
I20240501 14:04:25.821018 2068 raft_consensus.cc:2781] T 4c22d4541da544f5b812c6e5e8c2ea40 P 05f4ef3d126d43e9be32731e708ff76c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:04:25.821170 2068 raft_consensus.cc:482] T 4c22d4541da544f5b812c6e5e8c2ea40 P 05f4ef3d126d43e9be32731e708ff76c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:04:25.821283 2068 raft_consensus.cc:3037] T 4c22d4541da544f5b812c6e5e8c2ea40 P 05f4ef3d126d43e9be32731e708ff76c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:25.823218 2068 raft_consensus.cc:504] T 4c22d4541da544f5b812c6e5e8c2ea40 P 05f4ef3d126d43e9be32731e708ff76c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } }
I20240501 14:04:25.823655 2068 leader_election.cc:288] T 4c22d4541da544f5b812c6e5e8c2ea40 P 05f4ef3d126d43e9be32731e708ff76c [CANDIDATE]: Term 1 election: Requested vote from peers eb004c2ac9d244f89b61448584d4115b (127.29.124.131:40225)
I20240501 14:04:25.824110 1453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c22d4541da544f5b812c6e5e8c2ea40" candidate_uuid: "05f4ef3d126d43e9be32731e708ff76c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb004c2ac9d244f89b61448584d4115b"
I20240501 14:04:25.824262 1453 raft_consensus.cc:3037] T 4c22d4541da544f5b812c6e5e8c2ea40 P eb004c2ac9d244f89b61448584d4115b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:25.826273 1453 raft_consensus.cc:2454] T 4c22d4541da544f5b812c6e5e8c2ea40 P eb004c2ac9d244f89b61448584d4115b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05f4ef3d126d43e9be32731e708ff76c in term 1.
I20240501 14:04:25.826607 1259 leader_election.cc:302] T 4c22d4541da544f5b812c6e5e8c2ea40 P 05f4ef3d126d43e9be32731e708ff76c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 05f4ef3d126d43e9be32731e708ff76c, eb004c2ac9d244f89b61448584d4115b; no voters:
I20240501 14:04:25.826859 2068 raft_consensus.cc:2781] T 4c22d4541da544f5b812c6e5e8c2ea40 P 05f4ef3d126d43e9be32731e708ff76c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:04:25.827219 2068 raft_consensus.cc:686] T 4c22d4541da544f5b812c6e5e8c2ea40 P 05f4ef3d126d43e9be32731e708ff76c [term 1 LEADER]: Becoming Leader. State: Replica: 05f4ef3d126d43e9be32731e708ff76c, State: Running, Role: LEADER
I20240501 14:04:25.827629 2068 consensus_queue.cc:237] T 4c22d4541da544f5b812c6e5e8c2ea40 P 05f4ef3d126d43e9be32731e708ff76c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } }
I20240501 14:04:25.831501 1052 catalog_manager.cc:5484] T 4c22d4541da544f5b812c6e5e8c2ea40 P 05f4ef3d126d43e9be32731e708ff76c reported cstate change: term changed from 0 to 1, leader changed from <none> to 05f4ef3d126d43e9be32731e708ff76c (127.29.124.130). New cstate: current_term: 1 leader_uuid: "05f4ef3d126d43e9be32731e708ff76c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } health_report { overall_health: UNKNOWN } } }
I20240501 14:04:25.841356 2072 raft_consensus.cc:482] T 8979fd4d21814622ba687edb88fa7851 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:04:25.841580 2072 raft_consensus.cc:504] T 8979fd4d21814622ba687edb88fa7851 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } }
I20240501 14:04:25.843048 2072 leader_election.cc:288] T 8979fd4d21814622ba687edb88fa7851 P 5d45680f41c84ac39bd97c6f00ac2b06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05f4ef3d126d43e9be32731e708ff76c (127.29.124.130:41511)
I20240501 14:04:25.849056 1323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8979fd4d21814622ba687edb88fa7851" candidate_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4ef3d126d43e9be32731e708ff76c" is_pre_election: true
I20240501 14:04:25.849460 1323 raft_consensus.cc:2454] T 8979fd4d21814622ba687edb88fa7851 P 05f4ef3d126d43e9be32731e708ff76c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d45680f41c84ac39bd97c6f00ac2b06 in term 0.
I20240501 14:04:25.850025 1128 leader_election.cc:302] T 8979fd4d21814622ba687edb88fa7851 P 5d45680f41c84ac39bd97c6f00ac2b06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 05f4ef3d126d43e9be32731e708ff76c, 5d45680f41c84ac39bd97c6f00ac2b06; no voters:
I20240501 14:04:25.850390 2072 raft_consensus.cc:2781] T 8979fd4d21814622ba687edb88fa7851 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:04:25.850522 2072 raft_consensus.cc:482] T 8979fd4d21814622ba687edb88fa7851 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:04:25.850612 2072 raft_consensus.cc:3037] T 8979fd4d21814622ba687edb88fa7851 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:25.852563 2072 raft_consensus.cc:504] T 8979fd4d21814622ba687edb88fa7851 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } }
I20240501 14:04:25.853009 2072 leader_election.cc:288] T 8979fd4d21814622ba687edb88fa7851 P 5d45680f41c84ac39bd97c6f00ac2b06 [CANDIDATE]: Term 1 election: Requested vote from peers 05f4ef3d126d43e9be32731e708ff76c (127.29.124.130:41511)
I20240501 14:04:25.853482 1323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8979fd4d21814622ba687edb88fa7851" candidate_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4ef3d126d43e9be32731e708ff76c"
I20240501 14:04:25.853652 1323 raft_consensus.cc:3037] T 8979fd4d21814622ba687edb88fa7851 P 05f4ef3d126d43e9be32731e708ff76c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:25.855638 1323 raft_consensus.cc:2454] T 8979fd4d21814622ba687edb88fa7851 P 05f4ef3d126d43e9be32731e708ff76c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d45680f41c84ac39bd97c6f00ac2b06 in term 1.
I20240501 14:04:25.855988 1128 leader_election.cc:302] T 8979fd4d21814622ba687edb88fa7851 P 5d45680f41c84ac39bd97c6f00ac2b06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 05f4ef3d126d43e9be32731e708ff76c, 5d45680f41c84ac39bd97c6f00ac2b06; no voters:
I20240501 14:04:25.856267 2072 raft_consensus.cc:2781] T 8979fd4d21814622ba687edb88fa7851 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:04:25.856654 2072 raft_consensus.cc:686] T 8979fd4d21814622ba687edb88fa7851 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 LEADER]: Becoming Leader. State: Replica: 5d45680f41c84ac39bd97c6f00ac2b06, State: Running, Role: LEADER
I20240501 14:04:25.857168 2072 consensus_queue.cc:237] T 8979fd4d21814622ba687edb88fa7851 P 5d45680f41c84ac39bd97c6f00ac2b06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } }
I20240501 14:04:25.861243 1052 catalog_manager.cc:5484] T 8979fd4d21814622ba687edb88fa7851 P 5d45680f41c84ac39bd97c6f00ac2b06 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d45680f41c84ac39bd97c6f00ac2b06 (127.29.124.129). New cstate: current_term: 1 leader_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } health_report { overall_health: UNKNOWN } } }
W20240501 14:04:25.869019 1499 tablet.cc:2369] T 0f837cac32ad4aaa970124639138c2db P eb004c2ac9d244f89b61448584d4115b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 14:04:25.933636 1369 tablet.cc:2369] T 4c22d4541da544f5b812c6e5e8c2ea40 P 05f4ef3d126d43e9be32731e708ff76c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:04:26.079489 2072 raft_consensus.cc:482] T 0f837cac32ad4aaa970124639138c2db P 5d45680f41c84ac39bd97c6f00ac2b06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:04:26.079742 2072 raft_consensus.cc:504] T 0f837cac32ad4aaa970124639138c2db P 5d45680f41c84ac39bd97c6f00ac2b06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } }
I20240501 14:04:26.080330 2072 leader_election.cc:288] T 0f837cac32ad4aaa970124639138c2db P 5d45680f41c84ac39bd97c6f00ac2b06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb004c2ac9d244f89b61448584d4115b (127.29.124.131:40225)
I20240501 14:04:26.085332 1453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f837cac32ad4aaa970124639138c2db" candidate_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb004c2ac9d244f89b61448584d4115b" is_pre_election: true
I20240501 14:04:26.085556 1453 raft_consensus.cc:2454] T 0f837cac32ad4aaa970124639138c2db P eb004c2ac9d244f89b61448584d4115b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d45680f41c84ac39bd97c6f00ac2b06 in term 0.
I20240501 14:04:26.085922 1129 leader_election.cc:302] T 0f837cac32ad4aaa970124639138c2db P 5d45680f41c84ac39bd97c6f00ac2b06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5d45680f41c84ac39bd97c6f00ac2b06, eb004c2ac9d244f89b61448584d4115b; no voters:
I20240501 14:04:26.086215 2072 raft_consensus.cc:2781] T 0f837cac32ad4aaa970124639138c2db P 5d45680f41c84ac39bd97c6f00ac2b06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:04:26.086304 2072 raft_consensus.cc:482] T 0f837cac32ad4aaa970124639138c2db P 5d45680f41c84ac39bd97c6f00ac2b06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:04:26.086390 2072 raft_consensus.cc:3037] T 0f837cac32ad4aaa970124639138c2db P 5d45680f41c84ac39bd97c6f00ac2b06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:26.088439 2072 raft_consensus.cc:504] T 0f837cac32ad4aaa970124639138c2db P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } }
I20240501 14:04:26.089684 2072 leader_election.cc:288] T 0f837cac32ad4aaa970124639138c2db P 5d45680f41c84ac39bd97c6f00ac2b06 [CANDIDATE]: Term 1 election: Requested vote from peers eb004c2ac9d244f89b61448584d4115b (127.29.124.131:40225)
I20240501 14:04:26.091135 1453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f837cac32ad4aaa970124639138c2db" candidate_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb004c2ac9d244f89b61448584d4115b"
I20240501 14:04:26.091472 1453 raft_consensus.cc:3037] T 0f837cac32ad4aaa970124639138c2db P eb004c2ac9d244f89b61448584d4115b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:26.093341 1453 raft_consensus.cc:2454] T 0f837cac32ad4aaa970124639138c2db P eb004c2ac9d244f89b61448584d4115b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d45680f41c84ac39bd97c6f00ac2b06 in term 1.
I20240501 14:04:26.093673 1129 leader_election.cc:302] T 0f837cac32ad4aaa970124639138c2db P 5d45680f41c84ac39bd97c6f00ac2b06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5d45680f41c84ac39bd97c6f00ac2b06, eb004c2ac9d244f89b61448584d4115b; no voters:
I20240501 14:04:26.093932 2072 raft_consensus.cc:2781] T 0f837cac32ad4aaa970124639138c2db P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:04:26.094089 2072 raft_consensus.cc:686] T 0f837cac32ad4aaa970124639138c2db P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 LEADER]: Becoming Leader. State: Replica: 5d45680f41c84ac39bd97c6f00ac2b06, State: Running, Role: LEADER
I20240501 14:04:26.094329 2072 consensus_queue.cc:237] T 0f837cac32ad4aaa970124639138c2db P 5d45680f41c84ac39bd97c6f00ac2b06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } }
I20240501 14:04:26.096644 1052 catalog_manager.cc:5484] T 0f837cac32ad4aaa970124639138c2db P 5d45680f41c84ac39bd97c6f00ac2b06 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d45680f41c84ac39bd97c6f00ac2b06 (127.29.124.129). New cstate: current_term: 1 leader_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } health_report { overall_health: HEALTHY } } }
I20240501 14:04:26.323403 2068 consensus_queue.cc:1035] T 4c22d4541da544f5b812c6e5e8c2ea40 P 05f4ef3d126d43e9be32731e708ff76c [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:04:26.347806 2072 consensus_queue.cc:1035] T 8979fd4d21814622ba687edb88fa7851 P 5d45680f41c84ac39bd97c6f00ac2b06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:04:26.423722 2081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:04:26.424048 2081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:04:26.431447 2081 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
I20240501 14:04:26.585873 1453 raft_consensus.cc:1261] T 0f837cac32ad4aaa970124639138c2db P eb004c2ac9d244f89b61448584d4115b [term 1 FOLLOWER]: Refusing update from remote peer 5d45680f41c84ac39bd97c6f00ac2b06: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:04:26.586777 2072 consensus_queue.cc:1035] T 0f837cac32ad4aaa970124639138c2db P 5d45680f41c84ac39bd97c6f00ac2b06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 194.936 ms
time per row: 0.0974681 ms
I20240501 14:04:26.767009 1048 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50932:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:04:26.775051 1173 tablet_service.cc:1467] Processing CreateTablet for tablet ef358ca6b48646638ec351cb06cf704d (DEFAULT_TABLE table=rebalance_test_table_1 [id=423117c09096433e93fec4726745a428]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:26.775198 1303 tablet_service.cc:1467] Processing CreateTablet for tablet ef358ca6b48646638ec351cb06cf704d (DEFAULT_TABLE table=rebalance_test_table_1 [id=423117c09096433e93fec4726745a428]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:26.775573 1173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef358ca6b48646638ec351cb06cf704d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:26.775661 1303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef358ca6b48646638ec351cb06cf704d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:26.776226 1432 tablet_service.cc:1467] Processing CreateTablet for tablet 14e026c65736401fb3aec7e6462ae001 (DEFAULT_TABLE table=rebalance_test_table_1 [id=423117c09096433e93fec4726745a428]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:26.776353 1302 tablet_service.cc:1467] Processing CreateTablet for tablet 6a5325b7aa824831a5fa056fb4439110 (DEFAULT_TABLE table=rebalance_test_table_1 [id=423117c09096433e93fec4726745a428]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:26.776623 1432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14e026c65736401fb3aec7e6462ae001. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:26.776690 1302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a5325b7aa824831a5fa056fb4439110. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:26.777640 1433 tablet_service.cc:1467] Processing CreateTablet for tablet 6a5325b7aa824831a5fa056fb4439110 (DEFAULT_TABLE table=rebalance_test_table_1 [id=423117c09096433e93fec4726745a428]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:26.778012 1433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a5325b7aa824831a5fa056fb4439110. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:26.775492 1172 tablet_service.cc:1467] Processing CreateTablet for tablet 14e026c65736401fb3aec7e6462ae001 (DEFAULT_TABLE table=rebalance_test_table_1 [id=423117c09096433e93fec4726745a428]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:26.778995 1172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14e026c65736401fb3aec7e6462ae001. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:26.785737 2136 tablet_bootstrap.cc:492] T 14e026c65736401fb3aec7e6462ae001 P 5d45680f41c84ac39bd97c6f00ac2b06: Bootstrap starting.
I20240501 14:04:26.787698 2136 tablet_bootstrap.cc:654] T 14e026c65736401fb3aec7e6462ae001 P 5d45680f41c84ac39bd97c6f00ac2b06: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:26.788826 2137 tablet_bootstrap.cc:492] T 6a5325b7aa824831a5fa056fb4439110 P 05f4ef3d126d43e9be32731e708ff76c: Bootstrap starting.
I20240501 14:04:26.789305 2135 tablet_bootstrap.cc:492] T 6a5325b7aa824831a5fa056fb4439110 P eb004c2ac9d244f89b61448584d4115b: Bootstrap starting.
I20240501 14:04:26.790658 2137 tablet_bootstrap.cc:654] T 6a5325b7aa824831a5fa056fb4439110 P 05f4ef3d126d43e9be32731e708ff76c: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:26.790957 2135 tablet_bootstrap.cc:654] T 6a5325b7aa824831a5fa056fb4439110 P eb004c2ac9d244f89b61448584d4115b: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:26.791106 2136 tablet_bootstrap.cc:492] T 14e026c65736401fb3aec7e6462ae001 P 5d45680f41c84ac39bd97c6f00ac2b06: No bootstrap required, opened a new log
I20240501 14:04:26.791252 2136 ts_tablet_manager.cc:1367] T 14e026c65736401fb3aec7e6462ae001 P 5d45680f41c84ac39bd97c6f00ac2b06: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20240501 14:04:26.792022 2136 raft_consensus.cc:348] T 14e026c65736401fb3aec7e6462ae001 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } }
I20240501 14:04:26.792240 2137 tablet_bootstrap.cc:492] T 6a5325b7aa824831a5fa056fb4439110 P 05f4ef3d126d43e9be32731e708ff76c: No bootstrap required, opened a new log
I20240501 14:04:26.792265 2136 raft_consensus.cc:374] T 14e026c65736401fb3aec7e6462ae001 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:26.792371 2137 ts_tablet_manager.cc:1367] T 6a5325b7aa824831a5fa056fb4439110 P 05f4ef3d126d43e9be32731e708ff76c: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:04:26.792392 2136 raft_consensus.cc:729] T 14e026c65736401fb3aec7e6462ae001 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d45680f41c84ac39bd97c6f00ac2b06, State: Initialized, Role: FOLLOWER
I20240501 14:04:26.792644 2136 consensus_queue.cc:260] T 14e026c65736401fb3aec7e6462ae001 P 5d45680f41c84ac39bd97c6f00ac2b06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } }
I20240501 14:04:26.793076 2137 raft_consensus.cc:348] T 6a5325b7aa824831a5fa056fb4439110 P 05f4ef3d126d43e9be32731e708ff76c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } }
I20240501 14:04:26.793376 2137 raft_consensus.cc:374] T 6a5325b7aa824831a5fa056fb4439110 P 05f4ef3d126d43e9be32731e708ff76c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:26.793438 2136 ts_tablet_manager.cc:1398] T 14e026c65736401fb3aec7e6462ae001 P 5d45680f41c84ac39bd97c6f00ac2b06: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:04:26.793515 2137 raft_consensus.cc:729] T 6a5325b7aa824831a5fa056fb4439110 P 05f4ef3d126d43e9be32731e708ff76c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4ef3d126d43e9be32731e708ff76c, State: Initialized, Role: FOLLOWER
I20240501 14:04:26.793792 2136 tablet_bootstrap.cc:492] T ef358ca6b48646638ec351cb06cf704d P 5d45680f41c84ac39bd97c6f00ac2b06: Bootstrap starting.
I20240501 14:04:26.793802 2137 consensus_queue.cc:260] T 6a5325b7aa824831a5fa056fb4439110 P 05f4ef3d126d43e9be32731e708ff76c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } }
I20240501 14:04:26.794557 2137 ts_tablet_manager.cc:1398] T 6a5325b7aa824831a5fa056fb4439110 P 05f4ef3d126d43e9be32731e708ff76c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:04:26.794870 2137 tablet_bootstrap.cc:492] T ef358ca6b48646638ec351cb06cf704d P 05f4ef3d126d43e9be32731e708ff76c: Bootstrap starting.
I20240501 14:04:26.795332 2136 tablet_bootstrap.cc:654] T ef358ca6b48646638ec351cb06cf704d P 5d45680f41c84ac39bd97c6f00ac2b06: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:26.796389 2137 tablet_bootstrap.cc:654] T ef358ca6b48646638ec351cb06cf704d P 05f4ef3d126d43e9be32731e708ff76c: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:26.799537 2135 tablet_bootstrap.cc:492] T 6a5325b7aa824831a5fa056fb4439110 P eb004c2ac9d244f89b61448584d4115b: No bootstrap required, opened a new log
I20240501 14:04:26.799688 2135 ts_tablet_manager.cc:1367] T 6a5325b7aa824831a5fa056fb4439110 P eb004c2ac9d244f89b61448584d4115b: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.007s
I20240501 14:04:26.800311 2137 tablet_bootstrap.cc:492] T ef358ca6b48646638ec351cb06cf704d P 05f4ef3d126d43e9be32731e708ff76c: No bootstrap required, opened a new log
I20240501 14:04:26.800493 2137 ts_tablet_manager.cc:1367] T ef358ca6b48646638ec351cb06cf704d P 05f4ef3d126d43e9be32731e708ff76c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240501 14:04:26.800470 2135 raft_consensus.cc:348] T 6a5325b7aa824831a5fa056fb4439110 P eb004c2ac9d244f89b61448584d4115b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } }
I20240501 14:04:26.800670 2135 raft_consensus.cc:374] T 6a5325b7aa824831a5fa056fb4439110 P eb004c2ac9d244f89b61448584d4115b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:26.800756 2135 raft_consensus.cc:729] T 6a5325b7aa824831a5fa056fb4439110 P eb004c2ac9d244f89b61448584d4115b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb004c2ac9d244f89b61448584d4115b, State: Initialized, Role: FOLLOWER
I20240501 14:04:26.800953 2135 consensus_queue.cc:260] T 6a5325b7aa824831a5fa056fb4439110 P eb004c2ac9d244f89b61448584d4115b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } }
I20240501 14:04:26.801285 2137 raft_consensus.cc:348] T ef358ca6b48646638ec351cb06cf704d P 05f4ef3d126d43e9be32731e708ff76c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } }
I20240501 14:04:26.801499 2137 raft_consensus.cc:374] T ef358ca6b48646638ec351cb06cf704d P 05f4ef3d126d43e9be32731e708ff76c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:26.801496 2135 ts_tablet_manager.cc:1398] T 6a5325b7aa824831a5fa056fb4439110 P eb004c2ac9d244f89b61448584d4115b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:04:26.801563 2137 raft_consensus.cc:729] T ef358ca6b48646638ec351cb06cf704d P 05f4ef3d126d43e9be32731e708ff76c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4ef3d126d43e9be32731e708ff76c, State: Initialized, Role: FOLLOWER
I20240501 14:04:26.801761 2135 tablet_bootstrap.cc:492] T 14e026c65736401fb3aec7e6462ae001 P eb004c2ac9d244f89b61448584d4115b: Bootstrap starting.
I20240501 14:04:26.801780 2137 consensus_queue.cc:260] T ef358ca6b48646638ec351cb06cf704d P 05f4ef3d126d43e9be32731e708ff76c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } }
I20240501 14:04:26.802253 2137 ts_tablet_manager.cc:1398] T ef358ca6b48646638ec351cb06cf704d P 05f4ef3d126d43e9be32731e708ff76c: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:04:26.804090 2136 tablet_bootstrap.cc:492] T ef358ca6b48646638ec351cb06cf704d P 5d45680f41c84ac39bd97c6f00ac2b06: No bootstrap required, opened a new log
I20240501 14:04:26.804172 2135 tablet_bootstrap.cc:654] T 14e026c65736401fb3aec7e6462ae001 P eb004c2ac9d244f89b61448584d4115b: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:26.804396 2136 ts_tablet_manager.cc:1367] T ef358ca6b48646638ec351cb06cf704d P 5d45680f41c84ac39bd97c6f00ac2b06: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20240501 14:04:26.805625 2136 raft_consensus.cc:348] T ef358ca6b48646638ec351cb06cf704d P 5d45680f41c84ac39bd97c6f00ac2b06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } }
I20240501 14:04:26.805977 2136 raft_consensus.cc:374] T ef358ca6b48646638ec351cb06cf704d P 5d45680f41c84ac39bd97c6f00ac2b06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:26.806149 2136 raft_consensus.cc:729] T ef358ca6b48646638ec351cb06cf704d P 5d45680f41c84ac39bd97c6f00ac2b06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d45680f41c84ac39bd97c6f00ac2b06, State: Initialized, Role: FOLLOWER
I20240501 14:04:26.806617 2136 consensus_queue.cc:260] T ef358ca6b48646638ec351cb06cf704d P 5d45680f41c84ac39bd97c6f00ac2b06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } }
I20240501 14:04:26.807282 2136 ts_tablet_manager.cc:1398] T ef358ca6b48646638ec351cb06cf704d P 5d45680f41c84ac39bd97c6f00ac2b06: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:04:26.808738 2135 tablet_bootstrap.cc:492] T 14e026c65736401fb3aec7e6462ae001 P eb004c2ac9d244f89b61448584d4115b: No bootstrap required, opened a new log
I20240501 14:04:26.808907 2135 ts_tablet_manager.cc:1367] T 14e026c65736401fb3aec7e6462ae001 P eb004c2ac9d244f89b61448584d4115b: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20240501 14:04:26.809912 2135 raft_consensus.cc:348] T 14e026c65736401fb3aec7e6462ae001 P eb004c2ac9d244f89b61448584d4115b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } }
I20240501 14:04:26.810096 2135 raft_consensus.cc:374] T 14e026c65736401fb3aec7e6462ae001 P eb004c2ac9d244f89b61448584d4115b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:26.810168 2135 raft_consensus.cc:729] T 14e026c65736401fb3aec7e6462ae001 P eb004c2ac9d244f89b61448584d4115b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb004c2ac9d244f89b61448584d4115b, State: Initialized, Role: FOLLOWER
I20240501 14:04:26.810420 2135 consensus_queue.cc:260] T 14e026c65736401fb3aec7e6462ae001 P eb004c2ac9d244f89b61448584d4115b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } }
I20240501 14:04:26.810885 2135 ts_tablet_manager.cc:1398] T 14e026c65736401fb3aec7e6462ae001 P eb004c2ac9d244f89b61448584d4115b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:04:26.896265 2082 raft_consensus.cc:482] T 6a5325b7aa824831a5fa056fb4439110 P eb004c2ac9d244f89b61448584d4115b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:04:26.896476 2082 raft_consensus.cc:504] T 6a5325b7aa824831a5fa056fb4439110 P eb004c2ac9d244f89b61448584d4115b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } }
I20240501 14:04:26.897871 2082 leader_election.cc:288] T 6a5325b7aa824831a5fa056fb4439110 P eb004c2ac9d244f89b61448584d4115b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05f4ef3d126d43e9be32731e708ff76c (127.29.124.130:41511)
I20240501 14:04:26.899646 2082 raft_consensus.cc:482] T 14e026c65736401fb3aec7e6462ae001 P eb004c2ac9d244f89b61448584d4115b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:04:26.899788 2082 raft_consensus.cc:504] T 14e026c65736401fb3aec7e6462ae001 P eb004c2ac9d244f89b61448584d4115b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } }
I20240501 14:04:26.900324 2082 leader_election.cc:288] T 14e026c65736401fb3aec7e6462ae001 P eb004c2ac9d244f89b61448584d4115b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d45680f41c84ac39bd97c6f00ac2b06 (127.29.124.129:40925)
I20240501 14:04:26.904647 1323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a5325b7aa824831a5fa056fb4439110" candidate_uuid: "eb004c2ac9d244f89b61448584d4115b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4ef3d126d43e9be32731e708ff76c" is_pre_election: true
I20240501 14:04:26.904852 1323 raft_consensus.cc:2454] T 6a5325b7aa824831a5fa056fb4439110 P 05f4ef3d126d43e9be32731e708ff76c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb004c2ac9d244f89b61448584d4115b in term 0.
I20240501 14:04:26.905362 1388 leader_election.cc:302] T 6a5325b7aa824831a5fa056fb4439110 P eb004c2ac9d244f89b61448584d4115b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 05f4ef3d126d43e9be32731e708ff76c, eb004c2ac9d244f89b61448584d4115b; no voters:
I20240501 14:04:26.905776 2082 raft_consensus.cc:2781] T 6a5325b7aa824831a5fa056fb4439110 P eb004c2ac9d244f89b61448584d4115b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:04:26.905871 2082 raft_consensus.cc:482] T 6a5325b7aa824831a5fa056fb4439110 P eb004c2ac9d244f89b61448584d4115b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:04:26.905974 2082 raft_consensus.cc:3037] T 6a5325b7aa824831a5fa056fb4439110 P eb004c2ac9d244f89b61448584d4115b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:26.907271 1193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14e026c65736401fb3aec7e6462ae001" candidate_uuid: "eb004c2ac9d244f89b61448584d4115b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" is_pre_election: true
I20240501 14:04:26.907614 1193 raft_consensus.cc:2454] T 14e026c65736401fb3aec7e6462ae001 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb004c2ac9d244f89b61448584d4115b in term 0.
I20240501 14:04:26.908004 1386 leader_election.cc:302] T 14e026c65736401fb3aec7e6462ae001 P eb004c2ac9d244f89b61448584d4115b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5d45680f41c84ac39bd97c6f00ac2b06, eb004c2ac9d244f89b61448584d4115b; no voters:
I20240501 14:04:26.908107 2082 raft_consensus.cc:504] T 6a5325b7aa824831a5fa056fb4439110 P eb004c2ac9d244f89b61448584d4115b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } }
I20240501 14:04:26.908618 2146 raft_consensus.cc:2781] T 14e026c65736401fb3aec7e6462ae001 P eb004c2ac9d244f89b61448584d4115b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:04:26.908594 2082 leader_election.cc:288] T 6a5325b7aa824831a5fa056fb4439110 P eb004c2ac9d244f89b61448584d4115b [CANDIDATE]: Term 1 election: Requested vote from peers 05f4ef3d126d43e9be32731e708ff76c (127.29.124.130:41511)
I20240501 14:04:26.908725 2146 raft_consensus.cc:482] T 14e026c65736401fb3aec7e6462ae001 P eb004c2ac9d244f89b61448584d4115b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:04:26.908838 2146 raft_consensus.cc:3037] T 14e026c65736401fb3aec7e6462ae001 P eb004c2ac9d244f89b61448584d4115b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:26.909085 1323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a5325b7aa824831a5fa056fb4439110" candidate_uuid: "eb004c2ac9d244f89b61448584d4115b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4ef3d126d43e9be32731e708ff76c"
I20240501 14:04:26.909338 1323 raft_consensus.cc:3037] T 6a5325b7aa824831a5fa056fb4439110 P 05f4ef3d126d43e9be32731e708ff76c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:26.911540 2146 raft_consensus.cc:504] T 14e026c65736401fb3aec7e6462ae001 P eb004c2ac9d244f89b61448584d4115b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } }
I20240501 14:04:26.911928 1323 raft_consensus.cc:2454] T 6a5325b7aa824831a5fa056fb4439110 P 05f4ef3d126d43e9be32731e708ff76c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb004c2ac9d244f89b61448584d4115b in term 1.
I20240501 14:04:26.912056 2146 leader_election.cc:288] T 14e026c65736401fb3aec7e6462ae001 P eb004c2ac9d244f89b61448584d4115b [CANDIDATE]: Term 1 election: Requested vote from peers 5d45680f41c84ac39bd97c6f00ac2b06 (127.29.124.129:40925)
I20240501 14:04:26.912248 1388 leader_election.cc:302] T 6a5325b7aa824831a5fa056fb4439110 P eb004c2ac9d244f89b61448584d4115b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 05f4ef3d126d43e9be32731e708ff76c, eb004c2ac9d244f89b61448584d4115b; no voters:
I20240501 14:04:26.912477 2146 raft_consensus.cc:2781] T 6a5325b7aa824831a5fa056fb4439110 P eb004c2ac9d244f89b61448584d4115b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:04:26.912508 1193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14e026c65736401fb3aec7e6462ae001" candidate_uuid: "eb004c2ac9d244f89b61448584d4115b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d45680f41c84ac39bd97c6f00ac2b06"
I20240501 14:04:26.912635 2146 raft_consensus.cc:686] T 6a5325b7aa824831a5fa056fb4439110 P eb004c2ac9d244f89b61448584d4115b [term 1 LEADER]: Becoming Leader. State: Replica: eb004c2ac9d244f89b61448584d4115b, State: Running, Role: LEADER
I20240501 14:04:26.912693 1193 raft_consensus.cc:3037] T 14e026c65736401fb3aec7e6462ae001 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:26.913102 2146 consensus_queue.cc:237] T 6a5325b7aa824831a5fa056fb4439110 P eb004c2ac9d244f89b61448584d4115b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } }
I20240501 14:04:26.914990 1193 raft_consensus.cc:2454] T 14e026c65736401fb3aec7e6462ae001 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb004c2ac9d244f89b61448584d4115b in term 1.
I20240501 14:04:26.915351 1386 leader_election.cc:302] T 14e026c65736401fb3aec7e6462ae001 P eb004c2ac9d244f89b61448584d4115b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5d45680f41c84ac39bd97c6f00ac2b06, eb004c2ac9d244f89b61448584d4115b; no voters:
I20240501 14:04:26.916172 1048 catalog_manager.cc:5484] T 6a5325b7aa824831a5fa056fb4439110 P eb004c2ac9d244f89b61448584d4115b reported cstate change: term changed from 0 to 1, leader changed from <none> to eb004c2ac9d244f89b61448584d4115b (127.29.124.131). New cstate: current_term: 1 leader_uuid: "eb004c2ac9d244f89b61448584d4115b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } health_report { overall_health: HEALTHY } } }
I20240501 14:04:26.916950 2146 raft_consensus.cc:2781] T 14e026c65736401fb3aec7e6462ae001 P eb004c2ac9d244f89b61448584d4115b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:04:26.917085 2146 raft_consensus.cc:686] T 14e026c65736401fb3aec7e6462ae001 P eb004c2ac9d244f89b61448584d4115b [term 1 LEADER]: Becoming Leader. State: Replica: eb004c2ac9d244f89b61448584d4115b, State: Running, Role: LEADER
I20240501 14:04:26.917309 2146 consensus_queue.cc:237] T 14e026c65736401fb3aec7e6462ae001 P eb004c2ac9d244f89b61448584d4115b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } }
I20240501 14:04:26.920630 1048 catalog_manager.cc:5484] T 14e026c65736401fb3aec7e6462ae001 P eb004c2ac9d244f89b61448584d4115b reported cstate change: term changed from 0 to 1, leader changed from <none> to eb004c2ac9d244f89b61448584d4115b (127.29.124.131). New cstate: current_term: 1 leader_uuid: "eb004c2ac9d244f89b61448584d4115b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } health_report { overall_health: UNKNOWN } } }
I20240501 14:04:26.940563 2087 raft_consensus.cc:482] T ef358ca6b48646638ec351cb06cf704d P 05f4ef3d126d43e9be32731e708ff76c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:04:26.940711 2087 raft_consensus.cc:504] T ef358ca6b48646638ec351cb06cf704d P 05f4ef3d126d43e9be32731e708ff76c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } }
I20240501 14:04:26.941335 2087 leader_election.cc:288] T ef358ca6b48646638ec351cb06cf704d P 05f4ef3d126d43e9be32731e708ff76c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d45680f41c84ac39bd97c6f00ac2b06 (127.29.124.129:40925)
I20240501 14:04:26.947201 1193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef358ca6b48646638ec351cb06cf704d" candidate_uuid: "05f4ef3d126d43e9be32731e708ff76c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" is_pre_election: true
I20240501 14:04:26.947412 1193 raft_consensus.cc:2454] T ef358ca6b48646638ec351cb06cf704d P 5d45680f41c84ac39bd97c6f00ac2b06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05f4ef3d126d43e9be32731e708ff76c in term 0.
I20240501 14:04:26.947758 1256 leader_election.cc:302] T ef358ca6b48646638ec351cb06cf704d P 05f4ef3d126d43e9be32731e708ff76c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 05f4ef3d126d43e9be32731e708ff76c, 5d45680f41c84ac39bd97c6f00ac2b06; no voters:
I20240501 14:04:26.948029 2087 raft_consensus.cc:2781] T ef358ca6b48646638ec351cb06cf704d P 05f4ef3d126d43e9be32731e708ff76c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:04:26.948128 2087 raft_consensus.cc:482] T ef358ca6b48646638ec351cb06cf704d P 05f4ef3d126d43e9be32731e708ff76c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:04:26.948190 2087 raft_consensus.cc:3037] T ef358ca6b48646638ec351cb06cf704d P 05f4ef3d126d43e9be32731e708ff76c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:26.950059 2087 raft_consensus.cc:504] T ef358ca6b48646638ec351cb06cf704d P 05f4ef3d126d43e9be32731e708ff76c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } }
I20240501 14:04:26.950439 2087 leader_election.cc:288] T ef358ca6b48646638ec351cb06cf704d P 05f4ef3d126d43e9be32731e708ff76c [CANDIDATE]: Term 1 election: Requested vote from peers 5d45680f41c84ac39bd97c6f00ac2b06 (127.29.124.129:40925)
I20240501 14:04:26.950850 1193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef358ca6b48646638ec351cb06cf704d" candidate_uuid: "05f4ef3d126d43e9be32731e708ff76c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d45680f41c84ac39bd97c6f00ac2b06"
I20240501 14:04:26.951020 1193 raft_consensus.cc:3037] T ef358ca6b48646638ec351cb06cf704d P 5d45680f41c84ac39bd97c6f00ac2b06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:26.952754 1193 raft_consensus.cc:2454] T ef358ca6b48646638ec351cb06cf704d P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05f4ef3d126d43e9be32731e708ff76c in term 1.
I20240501 14:04:26.953079 1256 leader_election.cc:302] T ef358ca6b48646638ec351cb06cf704d P 05f4ef3d126d43e9be32731e708ff76c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 05f4ef3d126d43e9be32731e708ff76c, 5d45680f41c84ac39bd97c6f00ac2b06; no voters:
I20240501 14:04:26.953385 2087 raft_consensus.cc:2781] T ef358ca6b48646638ec351cb06cf704d P 05f4ef3d126d43e9be32731e708ff76c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:04:26.953507 2087 raft_consensus.cc:686] T ef358ca6b48646638ec351cb06cf704d P 05f4ef3d126d43e9be32731e708ff76c [term 1 LEADER]: Becoming Leader. State: Replica: 05f4ef3d126d43e9be32731e708ff76c, State: Running, Role: LEADER
I20240501 14:04:26.953769 2087 consensus_queue.cc:237] T ef358ca6b48646638ec351cb06cf704d P 05f4ef3d126d43e9be32731e708ff76c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } }
I20240501 14:04:26.956014 1048 catalog_manager.cc:5484] T ef358ca6b48646638ec351cb06cf704d P 05f4ef3d126d43e9be32731e708ff76c reported cstate change: term changed from 0 to 1, leader changed from <none> to 05f4ef3d126d43e9be32731e708ff76c (127.29.124.130). New cstate: current_term: 1 leader_uuid: "05f4ef3d126d43e9be32731e708ff76c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } health_report { overall_health: HEALTHY } } }
W20240501 14:04:27.290688 2151 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:04:27.290997 2151 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:04:27.298743 2151 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240501 14:04:27.355957 2146 consensus_queue.cc:1035] T 14e026c65736401fb3aec7e6462ae001 P eb004c2ac9d244f89b61448584d4115b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:04:27.375110 2082 consensus_queue.cc:1035] T 6a5325b7aa824831a5fa056fb4439110 P eb004c2ac9d244f89b61448584d4115b [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:04:27.483274 1193 raft_consensus.cc:1261] T ef358ca6b48646638ec351cb06cf704d P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 FOLLOWER]: Refusing update from remote peer 05f4ef3d126d43e9be32731e708ff76c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:04:27.483901 2087 consensus_queue.cc:1035] T ef358ca6b48646638ec351cb06cf704d P 05f4ef3d126d43e9be32731e708ff76c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 207.807 ms
time per row: 0.103903 ms
I20240501 14:04:27.640880 1050 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50932:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:04:27.649642 1172 tablet_service.cc:1467] Processing CreateTablet for tablet 3a4ca3f4edcd4bf29d0bab40a8856039 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9c1d190b2b264f27af23180ab8f30d42]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:27.650125 1172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a4ca3f4edcd4bf29d0bab40a8856039. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:27.650285 1433 tablet_service.cc:1467] Processing CreateTablet for tablet 70afcb72943b49a7bea962791b38a91d (DEFAULT_TABLE table=rebalance_test_table_2 [id=9c1d190b2b264f27af23180ab8f30d42]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:27.650389 1302 tablet_service.cc:1467] Processing CreateTablet for tablet 3a4ca3f4edcd4bf29d0bab40a8856039 (DEFAULT_TABLE table=rebalance_test_table_2 [id=9c1d190b2b264f27af23180ab8f30d42]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:27.650700 1433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70afcb72943b49a7bea962791b38a91d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:27.650748 1302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a4ca3f4edcd4bf29d0bab40a8856039. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:27.653612 1303 tablet_service.cc:1467] Processing CreateTablet for tablet 53249c71dab94ff4b179309088f4d1cd (DEFAULT_TABLE table=rebalance_test_table_2 [id=9c1d190b2b264f27af23180ab8f30d42]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:27.653892 1303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53249c71dab94ff4b179309088f4d1cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:27.656502 1432 tablet_service.cc:1467] Processing CreateTablet for tablet 53249c71dab94ff4b179309088f4d1cd (DEFAULT_TABLE table=rebalance_test_table_2 [id=9c1d190b2b264f27af23180ab8f30d42]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:27.656849 1432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53249c71dab94ff4b179309088f4d1cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:27.657900 1173 tablet_service.cc:1467] Processing CreateTablet for tablet 70afcb72943b49a7bea962791b38a91d (DEFAULT_TABLE table=rebalance_test_table_2 [id=9c1d190b2b264f27af23180ab8f30d42]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:27.658336 1173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70afcb72943b49a7bea962791b38a91d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:27.665825 2203 tablet_bootstrap.cc:492] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 05f4ef3d126d43e9be32731e708ff76c: Bootstrap starting.
I20240501 14:04:27.669899 2203 tablet_bootstrap.cc:654] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 05f4ef3d126d43e9be32731e708ff76c: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:27.670444 2202 tablet_bootstrap.cc:492] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 5d45680f41c84ac39bd97c6f00ac2b06: Bootstrap starting.
I20240501 14:04:27.670934 2204 tablet_bootstrap.cc:492] T 53249c71dab94ff4b179309088f4d1cd P eb004c2ac9d244f89b61448584d4115b: Bootstrap starting.
I20240501 14:04:27.672084 2202 tablet_bootstrap.cc:654] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 5d45680f41c84ac39bd97c6f00ac2b06: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:27.672700 2204 tablet_bootstrap.cc:654] T 53249c71dab94ff4b179309088f4d1cd P eb004c2ac9d244f89b61448584d4115b: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:27.673110 2203 tablet_bootstrap.cc:492] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 05f4ef3d126d43e9be32731e708ff76c: No bootstrap required, opened a new log
I20240501 14:04:27.673290 2203 ts_tablet_manager.cc:1367] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 05f4ef3d126d43e9be32731e708ff76c: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240501 14:04:27.674157 2202 tablet_bootstrap.cc:492] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 5d45680f41c84ac39bd97c6f00ac2b06: No bootstrap required, opened a new log
I20240501 14:04:27.674070 2203 raft_consensus.cc:348] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 05f4ef3d126d43e9be32731e708ff76c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } }
I20240501 14:04:27.674288 2202 ts_tablet_manager.cc:1367] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 5d45680f41c84ac39bd97c6f00ac2b06: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:04:27.674288 2203 raft_consensus.cc:374] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 05f4ef3d126d43e9be32731e708ff76c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:27.674376 2203 raft_consensus.cc:729] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 05f4ef3d126d43e9be32731e708ff76c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4ef3d126d43e9be32731e708ff76c, State: Initialized, Role: FOLLOWER
I20240501 14:04:27.674814 2203 consensus_queue.cc:260] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 05f4ef3d126d43e9be32731e708ff76c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } }
I20240501 14:04:27.674911 2202 raft_consensus.cc:348] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } }
I20240501 14:04:27.675081 2202 raft_consensus.cc:374] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:27.675168 2202 raft_consensus.cc:729] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d45680f41c84ac39bd97c6f00ac2b06, State: Initialized, Role: FOLLOWER
I20240501 14:04:27.675375 2202 consensus_queue.cc:260] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 5d45680f41c84ac39bd97c6f00ac2b06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } }
I20240501 14:04:27.675791 2203 ts_tablet_manager.cc:1398] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 05f4ef3d126d43e9be32731e708ff76c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:04:27.675853 2202 ts_tablet_manager.cc:1398] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 5d45680f41c84ac39bd97c6f00ac2b06: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 14:04:27.676057 2203 tablet_bootstrap.cc:492] T 53249c71dab94ff4b179309088f4d1cd P 05f4ef3d126d43e9be32731e708ff76c: Bootstrap starting.
I20240501 14:04:27.676107 2202 tablet_bootstrap.cc:492] T 70afcb72943b49a7bea962791b38a91d P 5d45680f41c84ac39bd97c6f00ac2b06: Bootstrap starting.
I20240501 14:04:27.677532 2202 tablet_bootstrap.cc:654] T 70afcb72943b49a7bea962791b38a91d P 5d45680f41c84ac39bd97c6f00ac2b06: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:27.677690 2203 tablet_bootstrap.cc:654] T 53249c71dab94ff4b179309088f4d1cd P 05f4ef3d126d43e9be32731e708ff76c: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:27.681437 2204 tablet_bootstrap.cc:492] T 53249c71dab94ff4b179309088f4d1cd P eb004c2ac9d244f89b61448584d4115b: No bootstrap required, opened a new log
I20240501 14:04:27.681589 2204 ts_tablet_manager.cc:1367] T 53249c71dab94ff4b179309088f4d1cd P eb004c2ac9d244f89b61448584d4115b: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.008s
I20240501 14:04:27.682368 2204 raft_consensus.cc:348] T 53249c71dab94ff4b179309088f4d1cd P eb004c2ac9d244f89b61448584d4115b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } }
I20240501 14:04:27.682571 2204 raft_consensus.cc:374] T 53249c71dab94ff4b179309088f4d1cd P eb004c2ac9d244f89b61448584d4115b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:27.682649 2204 raft_consensus.cc:729] T 53249c71dab94ff4b179309088f4d1cd P eb004c2ac9d244f89b61448584d4115b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb004c2ac9d244f89b61448584d4115b, State: Initialized, Role: FOLLOWER
I20240501 14:04:27.682890 2204 consensus_queue.cc:260] T 53249c71dab94ff4b179309088f4d1cd P eb004c2ac9d244f89b61448584d4115b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } }
I20240501 14:04:27.683439 2204 ts_tablet_manager.cc:1398] T 53249c71dab94ff4b179309088f4d1cd P eb004c2ac9d244f89b61448584d4115b: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:04:27.683732 2204 tablet_bootstrap.cc:492] T 70afcb72943b49a7bea962791b38a91d P eb004c2ac9d244f89b61448584d4115b: Bootstrap starting.
I20240501 14:04:27.685472 2204 tablet_bootstrap.cc:654] T 70afcb72943b49a7bea962791b38a91d P eb004c2ac9d244f89b61448584d4115b: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:27.686847 2202 tablet_bootstrap.cc:492] T 70afcb72943b49a7bea962791b38a91d P 5d45680f41c84ac39bd97c6f00ac2b06: No bootstrap required, opened a new log
I20240501 14:04:27.686995 2202 ts_tablet_manager.cc:1367] T 70afcb72943b49a7bea962791b38a91d P 5d45680f41c84ac39bd97c6f00ac2b06: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.007s
I20240501 14:04:27.687307 2203 tablet_bootstrap.cc:492] T 53249c71dab94ff4b179309088f4d1cd P 05f4ef3d126d43e9be32731e708ff76c: No bootstrap required, opened a new log
I20240501 14:04:27.687474 2203 ts_tablet_manager.cc:1367] T 53249c71dab94ff4b179309088f4d1cd P 05f4ef3d126d43e9be32731e708ff76c: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20240501 14:04:27.687690 2202 raft_consensus.cc:348] T 70afcb72943b49a7bea962791b38a91d P 5d45680f41c84ac39bd97c6f00ac2b06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } }
I20240501 14:04:27.687876 2202 raft_consensus.cc:374] T 70afcb72943b49a7bea962791b38a91d P 5d45680f41c84ac39bd97c6f00ac2b06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:27.687947 2202 raft_consensus.cc:729] T 70afcb72943b49a7bea962791b38a91d P 5d45680f41c84ac39bd97c6f00ac2b06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d45680f41c84ac39bd97c6f00ac2b06, State: Initialized, Role: FOLLOWER
I20240501 14:04:27.688164 2202 consensus_queue.cc:260] T 70afcb72943b49a7bea962791b38a91d P 5d45680f41c84ac39bd97c6f00ac2b06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } }
I20240501 14:04:27.688190 2203 raft_consensus.cc:348] T 53249c71dab94ff4b179309088f4d1cd P 05f4ef3d126d43e9be32731e708ff76c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } }
I20240501 14:04:27.688395 2203 raft_consensus.cc:374] T 53249c71dab94ff4b179309088f4d1cd P 05f4ef3d126d43e9be32731e708ff76c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:27.688458 2203 raft_consensus.cc:729] T 53249c71dab94ff4b179309088f4d1cd P 05f4ef3d126d43e9be32731e708ff76c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4ef3d126d43e9be32731e708ff76c, State: Initialized, Role: FOLLOWER
I20240501 14:04:27.688634 2202 ts_tablet_manager.cc:1398] T 70afcb72943b49a7bea962791b38a91d P 5d45680f41c84ac39bd97c6f00ac2b06: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:04:27.688711 2203 consensus_queue.cc:260] T 53249c71dab94ff4b179309088f4d1cd P 05f4ef3d126d43e9be32731e708ff76c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } }
I20240501 14:04:27.689288 2203 ts_tablet_manager.cc:1398] T 53249c71dab94ff4b179309088f4d1cd P 05f4ef3d126d43e9be32731e708ff76c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:04:27.690079 2204 tablet_bootstrap.cc:492] T 70afcb72943b49a7bea962791b38a91d P eb004c2ac9d244f89b61448584d4115b: No bootstrap required, opened a new log
I20240501 14:04:27.690228 2204 ts_tablet_manager.cc:1367] T 70afcb72943b49a7bea962791b38a91d P eb004c2ac9d244f89b61448584d4115b: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.001s
I20240501 14:04:27.690908 2204 raft_consensus.cc:348] T 70afcb72943b49a7bea962791b38a91d P eb004c2ac9d244f89b61448584d4115b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } }
I20240501 14:04:27.691669 2204 raft_consensus.cc:374] T 70afcb72943b49a7bea962791b38a91d P eb004c2ac9d244f89b61448584d4115b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:27.691766 2204 raft_consensus.cc:729] T 70afcb72943b49a7bea962791b38a91d P eb004c2ac9d244f89b61448584d4115b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb004c2ac9d244f89b61448584d4115b, State: Initialized, Role: FOLLOWER
I20240501 14:04:27.691979 2204 consensus_queue.cc:260] T 70afcb72943b49a7bea962791b38a91d P eb004c2ac9d244f89b61448584d4115b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } }
I20240501 14:04:27.692472 2204 ts_tablet_manager.cc:1398] T 70afcb72943b49a7bea962791b38a91d P eb004c2ac9d244f89b61448584d4115b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:04:27.729972 2195 raft_consensus.cc:482] T 53249c71dab94ff4b179309088f4d1cd P 05f4ef3d126d43e9be32731e708ff76c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:04:27.730141 2195 raft_consensus.cc:504] T 53249c71dab94ff4b179309088f4d1cd P 05f4ef3d126d43e9be32731e708ff76c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } }
I20240501 14:04:27.730672 2195 leader_election.cc:288] T 53249c71dab94ff4b179309088f4d1cd P 05f4ef3d126d43e9be32731e708ff76c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb004c2ac9d244f89b61448584d4115b (127.29.124.131:40225)
I20240501 14:04:27.731210 1452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53249c71dab94ff4b179309088f4d1cd" candidate_uuid: "05f4ef3d126d43e9be32731e708ff76c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb004c2ac9d244f89b61448584d4115b" is_pre_election: true
I20240501 14:04:27.731415 1452 raft_consensus.cc:2454] T 53249c71dab94ff4b179309088f4d1cd P eb004c2ac9d244f89b61448584d4115b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05f4ef3d126d43e9be32731e708ff76c in term 0.
I20240501 14:04:27.731827 1259 leader_election.cc:302] T 53249c71dab94ff4b179309088f4d1cd P 05f4ef3d126d43e9be32731e708ff76c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 05f4ef3d126d43e9be32731e708ff76c, eb004c2ac9d244f89b61448584d4115b; no voters:
I20240501 14:04:27.732080 2195 raft_consensus.cc:2781] T 53249c71dab94ff4b179309088f4d1cd P 05f4ef3d126d43e9be32731e708ff76c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:04:27.732185 2195 raft_consensus.cc:482] T 53249c71dab94ff4b179309088f4d1cd P 05f4ef3d126d43e9be32731e708ff76c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:04:27.732270 2195 raft_consensus.cc:3037] T 53249c71dab94ff4b179309088f4d1cd P 05f4ef3d126d43e9be32731e708ff76c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:27.734280 2195 raft_consensus.cc:504] T 53249c71dab94ff4b179309088f4d1cd P 05f4ef3d126d43e9be32731e708ff76c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } }
I20240501 14:04:27.734735 2195 leader_election.cc:288] T 53249c71dab94ff4b179309088f4d1cd P 05f4ef3d126d43e9be32731e708ff76c [CANDIDATE]: Term 1 election: Requested vote from peers eb004c2ac9d244f89b61448584d4115b (127.29.124.131:40225)
I20240501 14:04:27.735193 1452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53249c71dab94ff4b179309088f4d1cd" candidate_uuid: "05f4ef3d126d43e9be32731e708ff76c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb004c2ac9d244f89b61448584d4115b"
I20240501 14:04:27.735375 1452 raft_consensus.cc:3037] T 53249c71dab94ff4b179309088f4d1cd P eb004c2ac9d244f89b61448584d4115b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:27.737221 1452 raft_consensus.cc:2454] T 53249c71dab94ff4b179309088f4d1cd P eb004c2ac9d244f89b61448584d4115b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05f4ef3d126d43e9be32731e708ff76c in term 1.
I20240501 14:04:27.737633 1259 leader_election.cc:302] T 53249c71dab94ff4b179309088f4d1cd P 05f4ef3d126d43e9be32731e708ff76c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 05f4ef3d126d43e9be32731e708ff76c, eb004c2ac9d244f89b61448584d4115b; no voters:
I20240501 14:04:27.737898 2195 raft_consensus.cc:2781] T 53249c71dab94ff4b179309088f4d1cd P 05f4ef3d126d43e9be32731e708ff76c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:04:27.738023 2195 raft_consensus.cc:686] T 53249c71dab94ff4b179309088f4d1cd P 05f4ef3d126d43e9be32731e708ff76c [term 1 LEADER]: Becoming Leader. State: Replica: 05f4ef3d126d43e9be32731e708ff76c, State: Running, Role: LEADER
I20240501 14:04:27.738258 2195 consensus_queue.cc:237] T 53249c71dab94ff4b179309088f4d1cd P 05f4ef3d126d43e9be32731e708ff76c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } }
I20240501 14:04:27.740540 1050 catalog_manager.cc:5484] T 53249c71dab94ff4b179309088f4d1cd P 05f4ef3d126d43e9be32731e708ff76c reported cstate change: term changed from 0 to 1, leader changed from <none> to 05f4ef3d126d43e9be32731e708ff76c (127.29.124.130). New cstate: current_term: 1 leader_uuid: "05f4ef3d126d43e9be32731e708ff76c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } health_report { overall_health: UNKNOWN } } }
I20240501 14:04:27.770128 2195 raft_consensus.cc:482] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 05f4ef3d126d43e9be32731e708ff76c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:04:27.770301 2195 raft_consensus.cc:504] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 05f4ef3d126d43e9be32731e708ff76c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } }
I20240501 14:04:27.770730 2195 leader_election.cc:288] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 05f4ef3d126d43e9be32731e708ff76c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d45680f41c84ac39bd97c6f00ac2b06 (127.29.124.129:40925)
I20240501 14:04:27.771239 1193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4ca3f4edcd4bf29d0bab40a8856039" candidate_uuid: "05f4ef3d126d43e9be32731e708ff76c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" is_pre_election: true
I20240501 14:04:27.771392 2130 raft_consensus.cc:482] T 70afcb72943b49a7bea962791b38a91d P 5d45680f41c84ac39bd97c6f00ac2b06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:04:27.771476 1193 raft_consensus.cc:2454] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05f4ef3d126d43e9be32731e708ff76c in term 0.
I20240501 14:04:27.771548 2130 raft_consensus.cc:504] T 70afcb72943b49a7bea962791b38a91d P 5d45680f41c84ac39bd97c6f00ac2b06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } }
I20240501 14:04:27.771875 1256 leader_election.cc:302] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 05f4ef3d126d43e9be32731e708ff76c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 05f4ef3d126d43e9be32731e708ff76c, 5d45680f41c84ac39bd97c6f00ac2b06; no voters:
I20240501 14:04:27.772055 2130 leader_election.cc:288] T 70afcb72943b49a7bea962791b38a91d P 5d45680f41c84ac39bd97c6f00ac2b06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb004c2ac9d244f89b61448584d4115b (127.29.124.131:40225)
I20240501 14:04:27.772173 2195 raft_consensus.cc:2781] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 05f4ef3d126d43e9be32731e708ff76c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:04:27.772275 2195 raft_consensus.cc:482] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 05f4ef3d126d43e9be32731e708ff76c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:04:27.772387 2195 raft_consensus.cc:3037] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 05f4ef3d126d43e9be32731e708ff76c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:27.772586 1452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70afcb72943b49a7bea962791b38a91d" candidate_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb004c2ac9d244f89b61448584d4115b" is_pre_election: true
I20240501 14:04:27.772781 1452 raft_consensus.cc:2454] T 70afcb72943b49a7bea962791b38a91d P eb004c2ac9d244f89b61448584d4115b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d45680f41c84ac39bd97c6f00ac2b06 in term 0.
I20240501 14:04:27.773188 1129 leader_election.cc:302] T 70afcb72943b49a7bea962791b38a91d P 5d45680f41c84ac39bd97c6f00ac2b06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5d45680f41c84ac39bd97c6f00ac2b06, eb004c2ac9d244f89b61448584d4115b; no voters:
I20240501 14:04:27.773476 2130 raft_consensus.cc:2781] T 70afcb72943b49a7bea962791b38a91d P 5d45680f41c84ac39bd97c6f00ac2b06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:04:27.773586 2130 raft_consensus.cc:482] T 70afcb72943b49a7bea962791b38a91d P 5d45680f41c84ac39bd97c6f00ac2b06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:04:27.773687 2130 raft_consensus.cc:3037] T 70afcb72943b49a7bea962791b38a91d P 5d45680f41c84ac39bd97c6f00ac2b06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:27.774865 2195 raft_consensus.cc:504] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 05f4ef3d126d43e9be32731e708ff76c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } }
I20240501 14:04:27.775365 2195 leader_election.cc:288] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 05f4ef3d126d43e9be32731e708ff76c [CANDIDATE]: Term 1 election: Requested vote from peers 5d45680f41c84ac39bd97c6f00ac2b06 (127.29.124.129:40925)
I20240501 14:04:27.775743 1193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4ca3f4edcd4bf29d0bab40a8856039" candidate_uuid: "05f4ef3d126d43e9be32731e708ff76c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d45680f41c84ac39bd97c6f00ac2b06"
I20240501 14:04:27.775918 1193 raft_consensus.cc:3037] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:27.776154 2130 raft_consensus.cc:504] T 70afcb72943b49a7bea962791b38a91d P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } }
I20240501 14:04:27.776552 2130 leader_election.cc:288] T 70afcb72943b49a7bea962791b38a91d P 5d45680f41c84ac39bd97c6f00ac2b06 [CANDIDATE]: Term 1 election: Requested vote from peers eb004c2ac9d244f89b61448584d4115b (127.29.124.131:40225)
I20240501 14:04:27.776947 1452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "70afcb72943b49a7bea962791b38a91d" candidate_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb004c2ac9d244f89b61448584d4115b"
I20240501 14:04:27.777127 1452 raft_consensus.cc:3037] T 70afcb72943b49a7bea962791b38a91d P eb004c2ac9d244f89b61448584d4115b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:27.777899 1193 raft_consensus.cc:2454] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05f4ef3d126d43e9be32731e708ff76c in term 1.
I20240501 14:04:27.778229 1256 leader_election.cc:302] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 05f4ef3d126d43e9be32731e708ff76c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 05f4ef3d126d43e9be32731e708ff76c, 5d45680f41c84ac39bd97c6f00ac2b06; no voters:
I20240501 14:04:27.778481 2195 raft_consensus.cc:2781] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 05f4ef3d126d43e9be32731e708ff76c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:04:27.778656 2195 raft_consensus.cc:686] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 05f4ef3d126d43e9be32731e708ff76c [term 1 LEADER]: Becoming Leader. State: Replica: 05f4ef3d126d43e9be32731e708ff76c, State: Running, Role: LEADER
I20240501 14:04:27.778888 2195 consensus_queue.cc:237] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 05f4ef3d126d43e9be32731e708ff76c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } }
I20240501 14:04:27.779008 1452 raft_consensus.cc:2454] T 70afcb72943b49a7bea962791b38a91d P eb004c2ac9d244f89b61448584d4115b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d45680f41c84ac39bd97c6f00ac2b06 in term 1.
I20240501 14:04:27.779356 1129 leader_election.cc:302] T 70afcb72943b49a7bea962791b38a91d P 5d45680f41c84ac39bd97c6f00ac2b06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5d45680f41c84ac39bd97c6f00ac2b06, eb004c2ac9d244f89b61448584d4115b; no voters:
I20240501 14:04:27.779687 2130 raft_consensus.cc:2781] T 70afcb72943b49a7bea962791b38a91d P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:04:27.780115 2130 raft_consensus.cc:686] T 70afcb72943b49a7bea962791b38a91d P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 LEADER]: Becoming Leader. State: Replica: 5d45680f41c84ac39bd97c6f00ac2b06, State: Running, Role: LEADER
I20240501 14:04:27.780578 2130 consensus_queue.cc:237] T 70afcb72943b49a7bea962791b38a91d P 5d45680f41c84ac39bd97c6f00ac2b06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } }
I20240501 14:04:27.781433 1050 catalog_manager.cc:5484] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 05f4ef3d126d43e9be32731e708ff76c reported cstate change: term changed from 0 to 1, leader changed from <none> to 05f4ef3d126d43e9be32731e708ff76c (127.29.124.130). New cstate: current_term: 1 leader_uuid: "05f4ef3d126d43e9be32731e708ff76c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } health_report { overall_health: HEALTHY } } }
I20240501 14:04:27.782778 1052 catalog_manager.cc:5484] T 70afcb72943b49a7bea962791b38a91d P 5d45680f41c84ac39bd97c6f00ac2b06 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d45680f41c84ac39bd97c6f00ac2b06 (127.29.124.129). New cstate: current_term: 1 leader_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } health_report { overall_health: UNKNOWN } } }
W20240501 14:04:28.096401 2215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:04:28.096669 2215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:04:28.104272 2215 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240501 14:04:28.198295 2195 consensus_queue.cc:1035] T 53249c71dab94ff4b179309088f4d1cd P 05f4ef3d126d43e9be32731e708ff76c [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:04:28.232616 1452 raft_consensus.cc:1261] T 70afcb72943b49a7bea962791b38a91d P eb004c2ac9d244f89b61448584d4115b [term 1 FOLLOWER]: Refusing update from remote peer 5d45680f41c84ac39bd97c6f00ac2b06: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:04:28.233816 2130 consensus_queue.cc:1035] T 70afcb72943b49a7bea962791b38a91d P 5d45680f41c84ac39bd97c6f00ac2b06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:04:28.235541 1193 raft_consensus.cc:1261] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 FOLLOWER]: Refusing update from remote peer 05f4ef3d126d43e9be32731e708ff76c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:04:28.236199 2192 consensus_queue.cc:1035] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 05f4ef3d126d43e9be32731e708ff76c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:04:28.255653 2250 mvcc.cc:205] Tried to move back new op lower bound from 7022888010688012288 to 7022888008823775232. Current Snapshot: MvccSnapshot[applied={T|T < 7022888010688012288}]
I20240501 14:04:28.256723 2245 mvcc.cc:205] Tried to move back new op lower bound from 7022888010688012288 to 7022888008823775232. Current Snapshot: MvccSnapshot[applied={T|T < 7022888010688012288}]
INSERT report
rows total: 2000
time total: 174.55 ms
time per row: 0.087275 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240501 14:04:29.270697 1051 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50932:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:04:29.280018 1302 tablet_service.cc:1467] Processing CreateTablet for tablet 1074c96bc2564edca376f95bd195a62c (DEFAULT_TABLE table=rebalance_test_table_3 [id=16c6cbe8fef84c88b36c2d1809ebe48b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:29.280248 1172 tablet_service.cc:1467] Processing CreateTablet for tablet 1074c96bc2564edca376f95bd195a62c (DEFAULT_TABLE table=rebalance_test_table_3 [id=16c6cbe8fef84c88b36c2d1809ebe48b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:29.280499 1302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1074c96bc2564edca376f95bd195a62c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:29.280642 1172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1074c96bc2564edca376f95bd195a62c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:29.281311 1433 tablet_service.cc:1467] Processing CreateTablet for tablet 14e0e4808438484688605ff2d6d9258b (DEFAULT_TABLE table=rebalance_test_table_3 [id=16c6cbe8fef84c88b36c2d1809ebe48b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:29.281692 1433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14e0e4808438484688605ff2d6d9258b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:29.282188 1432 tablet_service.cc:1467] Processing CreateTablet for tablet 0f11914838ca4139a1c1813fc5334782 (DEFAULT_TABLE table=rebalance_test_table_3 [id=16c6cbe8fef84c88b36c2d1809ebe48b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:29.282586 1432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f11914838ca4139a1c1813fc5334782. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:29.286729 1303 tablet_service.cc:1467] Processing CreateTablet for tablet 14e0e4808438484688605ff2d6d9258b (DEFAULT_TABLE table=rebalance_test_table_3 [id=16c6cbe8fef84c88b36c2d1809ebe48b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:29.287086 1303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14e0e4808438484688605ff2d6d9258b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:29.290747 2273 tablet_bootstrap.cc:492] T 14e0e4808438484688605ff2d6d9258b P eb004c2ac9d244f89b61448584d4115b: Bootstrap starting.
I20240501 14:04:29.293040 2273 tablet_bootstrap.cc:654] T 14e0e4808438484688605ff2d6d9258b P eb004c2ac9d244f89b61448584d4115b: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:29.293186 1173 tablet_service.cc:1467] Processing CreateTablet for tablet 0f11914838ca4139a1c1813fc5334782 (DEFAULT_TABLE table=rebalance_test_table_3 [id=16c6cbe8fef84c88b36c2d1809ebe48b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:29.293606 1173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f11914838ca4139a1c1813fc5334782. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:29.295253 2273 tablet_bootstrap.cc:492] T 14e0e4808438484688605ff2d6d9258b P eb004c2ac9d244f89b61448584d4115b: No bootstrap required, opened a new log
I20240501 14:04:29.295406 2273 ts_tablet_manager.cc:1367] T 14e0e4808438484688605ff2d6d9258b P eb004c2ac9d244f89b61448584d4115b: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240501 14:04:29.296173 2273 raft_consensus.cc:348] T 14e0e4808438484688605ff2d6d9258b P eb004c2ac9d244f89b61448584d4115b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } }
I20240501 14:04:29.296393 2273 raft_consensus.cc:374] T 14e0e4808438484688605ff2d6d9258b P eb004c2ac9d244f89b61448584d4115b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:29.296484 2273 raft_consensus.cc:729] T 14e0e4808438484688605ff2d6d9258b P eb004c2ac9d244f89b61448584d4115b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb004c2ac9d244f89b61448584d4115b, State: Initialized, Role: FOLLOWER
I20240501 14:04:29.298138 2273 consensus_queue.cc:260] T 14e0e4808438484688605ff2d6d9258b P eb004c2ac9d244f89b61448584d4115b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } }
I20240501 14:04:29.298658 2272 tablet_bootstrap.cc:492] T 1074c96bc2564edca376f95bd195a62c P 5d45680f41c84ac39bd97c6f00ac2b06: Bootstrap starting.
I20240501 14:04:29.298691 2273 ts_tablet_manager.cc:1398] T 14e0e4808438484688605ff2d6d9258b P eb004c2ac9d244f89b61448584d4115b: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:04:29.298980 2273 tablet_bootstrap.cc:492] T 0f11914838ca4139a1c1813fc5334782 P eb004c2ac9d244f89b61448584d4115b: Bootstrap starting.
I20240501 14:04:29.300194 2272 tablet_bootstrap.cc:654] T 1074c96bc2564edca376f95bd195a62c P 5d45680f41c84ac39bd97c6f00ac2b06: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:29.301229 2273 tablet_bootstrap.cc:654] T 0f11914838ca4139a1c1813fc5334782 P eb004c2ac9d244f89b61448584d4115b: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:29.301434 2271 tablet_bootstrap.cc:492] T 1074c96bc2564edca376f95bd195a62c P 05f4ef3d126d43e9be32731e708ff76c: Bootstrap starting.
I20240501 14:04:29.303004 2271 tablet_bootstrap.cc:654] T 1074c96bc2564edca376f95bd195a62c P 05f4ef3d126d43e9be32731e708ff76c: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:29.303238 2273 tablet_bootstrap.cc:492] T 0f11914838ca4139a1c1813fc5334782 P eb004c2ac9d244f89b61448584d4115b: No bootstrap required, opened a new log
I20240501 14:04:29.303256 2272 tablet_bootstrap.cc:492] T 1074c96bc2564edca376f95bd195a62c P 5d45680f41c84ac39bd97c6f00ac2b06: No bootstrap required, opened a new log
I20240501 14:04:29.303388 2272 ts_tablet_manager.cc:1367] T 1074c96bc2564edca376f95bd195a62c P 5d45680f41c84ac39bd97c6f00ac2b06: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 14:04:29.303455 2273 ts_tablet_manager.cc:1367] T 0f11914838ca4139a1c1813fc5334782 P eb004c2ac9d244f89b61448584d4115b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 14:04:29.304131 2272 raft_consensus.cc:348] T 1074c96bc2564edca376f95bd195a62c P 5d45680f41c84ac39bd97c6f00ac2b06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } }
I20240501 14:04:29.304244 2273 raft_consensus.cc:348] T 0f11914838ca4139a1c1813fc5334782 P eb004c2ac9d244f89b61448584d4115b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } }
I20240501 14:04:29.304409 2272 raft_consensus.cc:374] T 1074c96bc2564edca376f95bd195a62c P 5d45680f41c84ac39bd97c6f00ac2b06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:29.304488 2273 raft_consensus.cc:374] T 0f11914838ca4139a1c1813fc5334782 P eb004c2ac9d244f89b61448584d4115b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:29.304512 2272 raft_consensus.cc:729] T 1074c96bc2564edca376f95bd195a62c P 5d45680f41c84ac39bd97c6f00ac2b06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d45680f41c84ac39bd97c6f00ac2b06, State: Initialized, Role: FOLLOWER
I20240501 14:04:29.304602 2273 raft_consensus.cc:729] T 0f11914838ca4139a1c1813fc5334782 P eb004c2ac9d244f89b61448584d4115b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb004c2ac9d244f89b61448584d4115b, State: Initialized, Role: FOLLOWER
I20240501 14:04:29.304755 2272 consensus_queue.cc:260] T 1074c96bc2564edca376f95bd195a62c P 5d45680f41c84ac39bd97c6f00ac2b06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } }
I20240501 14:04:29.304838 2273 consensus_queue.cc:260] T 0f11914838ca4139a1c1813fc5334782 P eb004c2ac9d244f89b61448584d4115b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } }
I20240501 14:04:29.305101 2271 tablet_bootstrap.cc:492] T 1074c96bc2564edca376f95bd195a62c P 05f4ef3d126d43e9be32731e708ff76c: No bootstrap required, opened a new log
I20240501 14:04:29.305310 2271 ts_tablet_manager.cc:1367] T 1074c96bc2564edca376f95bd195a62c P 05f4ef3d126d43e9be32731e708ff76c: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240501 14:04:29.305356 2273 ts_tablet_manager.cc:1398] T 0f11914838ca4139a1c1813fc5334782 P eb004c2ac9d244f89b61448584d4115b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:04:29.305406 2272 ts_tablet_manager.cc:1398] T 1074c96bc2564edca376f95bd195a62c P 5d45680f41c84ac39bd97c6f00ac2b06: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:04:29.305714 2272 tablet_bootstrap.cc:492] T 0f11914838ca4139a1c1813fc5334782 P 5d45680f41c84ac39bd97c6f00ac2b06: Bootstrap starting.
I20240501 14:04:29.306154 2271 raft_consensus.cc:348] T 1074c96bc2564edca376f95bd195a62c P 05f4ef3d126d43e9be32731e708ff76c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } }
I20240501 14:04:29.306407 2271 raft_consensus.cc:374] T 1074c96bc2564edca376f95bd195a62c P 05f4ef3d126d43e9be32731e708ff76c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:29.306543 2271 raft_consensus.cc:729] T 1074c96bc2564edca376f95bd195a62c P 05f4ef3d126d43e9be32731e708ff76c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4ef3d126d43e9be32731e708ff76c, State: Initialized, Role: FOLLOWER
I20240501 14:04:29.306833 2271 consensus_queue.cc:260] T 1074c96bc2564edca376f95bd195a62c P 05f4ef3d126d43e9be32731e708ff76c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } }
I20240501 14:04:29.307541 2271 ts_tablet_manager.cc:1398] T 1074c96bc2564edca376f95bd195a62c P 05f4ef3d126d43e9be32731e708ff76c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:04:29.307737 2272 tablet_bootstrap.cc:654] T 0f11914838ca4139a1c1813fc5334782 P 5d45680f41c84ac39bd97c6f00ac2b06: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:29.307863 2271 tablet_bootstrap.cc:492] T 14e0e4808438484688605ff2d6d9258b P 05f4ef3d126d43e9be32731e708ff76c: Bootstrap starting.
I20240501 14:04:29.308602 2178 raft_consensus.cc:482] T 0f11914838ca4139a1c1813fc5334782 P eb004c2ac9d244f89b61448584d4115b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:04:29.308746 2178 raft_consensus.cc:504] T 0f11914838ca4139a1c1813fc5334782 P eb004c2ac9d244f89b61448584d4115b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } }
I20240501 14:04:29.309299 2178 leader_election.cc:288] T 0f11914838ca4139a1c1813fc5334782 P eb004c2ac9d244f89b61448584d4115b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d45680f41c84ac39bd97c6f00ac2b06 (127.29.124.129:40925)
I20240501 14:04:29.309402 2271 tablet_bootstrap.cc:654] T 14e0e4808438484688605ff2d6d9258b P 05f4ef3d126d43e9be32731e708ff76c: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:29.309948 1193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f11914838ca4139a1c1813fc5334782" candidate_uuid: "eb004c2ac9d244f89b61448584d4115b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" is_pre_election: true
W20240501 14:04:29.310839 1386 leader_election.cc:341] T 0f11914838ca4139a1c1813fc5334782 P eb004c2ac9d244f89b61448584d4115b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5d45680f41c84ac39bd97c6f00ac2b06 (127.29.124.129:40925): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:04:29.311046 1386 leader_election.cc:302] T 0f11914838ca4139a1c1813fc5334782 P eb004c2ac9d244f89b61448584d4115b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: eb004c2ac9d244f89b61448584d4115b; no voters: 5d45680f41c84ac39bd97c6f00ac2b06
I20240501 14:04:29.311782 2178 raft_consensus.cc:2726] T 0f11914838ca4139a1c1813fc5334782 P eb004c2ac9d244f89b61448584d4115b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:04:29.313565 2271 tablet_bootstrap.cc:492] T 14e0e4808438484688605ff2d6d9258b P 05f4ef3d126d43e9be32731e708ff76c: No bootstrap required, opened a new log
I20240501 14:04:29.313716 2271 ts_tablet_manager.cc:1367] T 14e0e4808438484688605ff2d6d9258b P 05f4ef3d126d43e9be32731e708ff76c: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20240501 14:04:29.313952 2272 tablet_bootstrap.cc:492] T 0f11914838ca4139a1c1813fc5334782 P 5d45680f41c84ac39bd97c6f00ac2b06: No bootstrap required, opened a new log
I20240501 14:04:29.314087 2272 ts_tablet_manager.cc:1367] T 0f11914838ca4139a1c1813fc5334782 P 5d45680f41c84ac39bd97c6f00ac2b06: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20240501 14:04:29.314376 2271 raft_consensus.cc:348] T 14e0e4808438484688605ff2d6d9258b P 05f4ef3d126d43e9be32731e708ff76c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } }
I20240501 14:04:29.314541 2271 raft_consensus.cc:374] T 14e0e4808438484688605ff2d6d9258b P 05f4ef3d126d43e9be32731e708ff76c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:29.314607 2271 raft_consensus.cc:729] T 14e0e4808438484688605ff2d6d9258b P 05f4ef3d126d43e9be32731e708ff76c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4ef3d126d43e9be32731e708ff76c, State: Initialized, Role: FOLLOWER
I20240501 14:04:29.314662 2272 raft_consensus.cc:348] T 0f11914838ca4139a1c1813fc5334782 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } }
I20240501 14:04:29.314837 2272 raft_consensus.cc:374] T 0f11914838ca4139a1c1813fc5334782 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:29.314819 2271 consensus_queue.cc:260] T 14e0e4808438484688605ff2d6d9258b P 05f4ef3d126d43e9be32731e708ff76c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } }
I20240501 14:04:29.314916 2272 raft_consensus.cc:729] T 0f11914838ca4139a1c1813fc5334782 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d45680f41c84ac39bd97c6f00ac2b06, State: Initialized, Role: FOLLOWER
I20240501 14:04:29.315120 2272 consensus_queue.cc:260] T 0f11914838ca4139a1c1813fc5334782 P 5d45680f41c84ac39bd97c6f00ac2b06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } }
I20240501 14:04:29.315313 2271 ts_tablet_manager.cc:1398] T 14e0e4808438484688605ff2d6d9258b P 05f4ef3d126d43e9be32731e708ff76c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:04:29.315579 2272 ts_tablet_manager.cc:1398] T 0f11914838ca4139a1c1813fc5334782 P 5d45680f41c84ac39bd97c6f00ac2b06: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 14:04:29.387660 2253 raft_consensus.cc:482] T 14e0e4808438484688605ff2d6d9258b P 05f4ef3d126d43e9be32731e708ff76c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:04:29.387838 2253 raft_consensus.cc:504] T 14e0e4808438484688605ff2d6d9258b P 05f4ef3d126d43e9be32731e708ff76c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } }
I20240501 14:04:29.388319 2253 leader_election.cc:288] T 14e0e4808438484688605ff2d6d9258b P 05f4ef3d126d43e9be32731e708ff76c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb004c2ac9d244f89b61448584d4115b (127.29.124.131:40225)
I20240501 14:04:29.388743 1452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14e0e4808438484688605ff2d6d9258b" candidate_uuid: "05f4ef3d126d43e9be32731e708ff76c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb004c2ac9d244f89b61448584d4115b" is_pre_election: true
I20240501 14:04:29.388933 1452 raft_consensus.cc:2454] T 14e0e4808438484688605ff2d6d9258b P eb004c2ac9d244f89b61448584d4115b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05f4ef3d126d43e9be32731e708ff76c in term 0.
I20240501 14:04:29.389331 1259 leader_election.cc:302] T 14e0e4808438484688605ff2d6d9258b P 05f4ef3d126d43e9be32731e708ff76c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 05f4ef3d126d43e9be32731e708ff76c, eb004c2ac9d244f89b61448584d4115b; no voters:
I20240501 14:04:29.389600 2253 raft_consensus.cc:2781] T 14e0e4808438484688605ff2d6d9258b P 05f4ef3d126d43e9be32731e708ff76c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:04:29.389695 2253 raft_consensus.cc:482] T 14e0e4808438484688605ff2d6d9258b P 05f4ef3d126d43e9be32731e708ff76c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:04:29.389780 2253 raft_consensus.cc:3037] T 14e0e4808438484688605ff2d6d9258b P 05f4ef3d126d43e9be32731e708ff76c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:29.391752 2253 raft_consensus.cc:504] T 14e0e4808438484688605ff2d6d9258b P 05f4ef3d126d43e9be32731e708ff76c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } }
I20240501 14:04:29.392148 2253 leader_election.cc:288] T 14e0e4808438484688605ff2d6d9258b P 05f4ef3d126d43e9be32731e708ff76c [CANDIDATE]: Term 1 election: Requested vote from peers eb004c2ac9d244f89b61448584d4115b (127.29.124.131:40225)
I20240501 14:04:29.392578 1452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "14e0e4808438484688605ff2d6d9258b" candidate_uuid: "05f4ef3d126d43e9be32731e708ff76c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb004c2ac9d244f89b61448584d4115b"
I20240501 14:04:29.392740 1452 raft_consensus.cc:3037] T 14e0e4808438484688605ff2d6d9258b P eb004c2ac9d244f89b61448584d4115b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:29.394510 1452 raft_consensus.cc:2454] T 14e0e4808438484688605ff2d6d9258b P eb004c2ac9d244f89b61448584d4115b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05f4ef3d126d43e9be32731e708ff76c in term 1.
I20240501 14:04:29.394867 1259 leader_election.cc:302] T 14e0e4808438484688605ff2d6d9258b P 05f4ef3d126d43e9be32731e708ff76c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 05f4ef3d126d43e9be32731e708ff76c, eb004c2ac9d244f89b61448584d4115b; no voters:
I20240501 14:04:29.395128 2253 raft_consensus.cc:2781] T 14e0e4808438484688605ff2d6d9258b P 05f4ef3d126d43e9be32731e708ff76c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:04:29.395439 2253 raft_consensus.cc:686] T 14e0e4808438484688605ff2d6d9258b P 05f4ef3d126d43e9be32731e708ff76c [term 1 LEADER]: Becoming Leader. State: Replica: 05f4ef3d126d43e9be32731e708ff76c, State: Running, Role: LEADER
I20240501 14:04:29.395665 2253 consensus_queue.cc:237] T 14e0e4808438484688605ff2d6d9258b P 05f4ef3d126d43e9be32731e708ff76c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } }
I20240501 14:04:29.397814 1052 catalog_manager.cc:5484] T 14e0e4808438484688605ff2d6d9258b P 05f4ef3d126d43e9be32731e708ff76c reported cstate change: term changed from 0 to 1, leader changed from <none> to 05f4ef3d126d43e9be32731e708ff76c (127.29.124.130). New cstate: current_term: 1 leader_uuid: "05f4ef3d126d43e9be32731e708ff76c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } health_report { overall_health: HEALTHY } } }
I20240501 14:04:29.596110 2262 raft_consensus.cc:482] T 0f11914838ca4139a1c1813fc5334782 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:04:29.596320 2262 raft_consensus.cc:504] T 0f11914838ca4139a1c1813fc5334782 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } }
I20240501 14:04:29.596881 2262 leader_election.cc:288] T 0f11914838ca4139a1c1813fc5334782 P 5d45680f41c84ac39bd97c6f00ac2b06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb004c2ac9d244f89b61448584d4115b (127.29.124.131:40225)
I20240501 14:04:29.597636 1452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f11914838ca4139a1c1813fc5334782" candidate_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb004c2ac9d244f89b61448584d4115b" is_pre_election: true
I20240501 14:04:29.597883 1452 raft_consensus.cc:2454] T 0f11914838ca4139a1c1813fc5334782 P eb004c2ac9d244f89b61448584d4115b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d45680f41c84ac39bd97c6f00ac2b06 in term 0.
I20240501 14:04:29.598337 1129 leader_election.cc:302] T 0f11914838ca4139a1c1813fc5334782 P 5d45680f41c84ac39bd97c6f00ac2b06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5d45680f41c84ac39bd97c6f00ac2b06, eb004c2ac9d244f89b61448584d4115b; no voters:
I20240501 14:04:29.598634 2262 raft_consensus.cc:2781] T 0f11914838ca4139a1c1813fc5334782 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:04:29.598747 2262 raft_consensus.cc:482] T 0f11914838ca4139a1c1813fc5334782 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:04:29.598835 2262 raft_consensus.cc:3037] T 0f11914838ca4139a1c1813fc5334782 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:29.601626 2262 raft_consensus.cc:504] T 0f11914838ca4139a1c1813fc5334782 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } }
I20240501 14:04:29.602108 2262 leader_election.cc:288] T 0f11914838ca4139a1c1813fc5334782 P 5d45680f41c84ac39bd97c6f00ac2b06 [CANDIDATE]: Term 1 election: Requested vote from peers eb004c2ac9d244f89b61448584d4115b (127.29.124.131:40225)
I20240501 14:04:29.602569 1452 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f11914838ca4139a1c1813fc5334782" candidate_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb004c2ac9d244f89b61448584d4115b"
I20240501 14:04:29.602766 1452 raft_consensus.cc:3037] T 0f11914838ca4139a1c1813fc5334782 P eb004c2ac9d244f89b61448584d4115b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:29.605116 1452 raft_consensus.cc:2454] T 0f11914838ca4139a1c1813fc5334782 P eb004c2ac9d244f89b61448584d4115b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d45680f41c84ac39bd97c6f00ac2b06 in term 1.
I20240501 14:04:29.605504 1129 leader_election.cc:302] T 0f11914838ca4139a1c1813fc5334782 P 5d45680f41c84ac39bd97c6f00ac2b06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5d45680f41c84ac39bd97c6f00ac2b06, eb004c2ac9d244f89b61448584d4115b; no voters:
I20240501 14:04:29.605791 2262 raft_consensus.cc:2781] T 0f11914838ca4139a1c1813fc5334782 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:04:29.606102 2262 raft_consensus.cc:686] T 0f11914838ca4139a1c1813fc5334782 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 LEADER]: Becoming Leader. State: Replica: 5d45680f41c84ac39bd97c6f00ac2b06, State: Running, Role: LEADER
I20240501 14:04:29.606405 2262 consensus_queue.cc:237] T 0f11914838ca4139a1c1813fc5334782 P 5d45680f41c84ac39bd97c6f00ac2b06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } }
I20240501 14:04:29.608613 1052 catalog_manager.cc:5484] T 0f11914838ca4139a1c1813fc5334782 P 5d45680f41c84ac39bd97c6f00ac2b06 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d45680f41c84ac39bd97c6f00ac2b06 (127.29.124.129). New cstate: current_term: 1 leader_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } health_report { overall_health: HEALTHY } } }
I20240501 14:04:29.721091 2262 raft_consensus.cc:482] T 1074c96bc2564edca376f95bd195a62c P 5d45680f41c84ac39bd97c6f00ac2b06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:04:29.721330 2262 raft_consensus.cc:504] T 1074c96bc2564edca376f95bd195a62c P 5d45680f41c84ac39bd97c6f00ac2b06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } }
I20240501 14:04:29.721755 2262 leader_election.cc:288] T 1074c96bc2564edca376f95bd195a62c P 5d45680f41c84ac39bd97c6f00ac2b06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05f4ef3d126d43e9be32731e708ff76c (127.29.124.130:41511)
I20240501 14:04:29.722222 1323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1074c96bc2564edca376f95bd195a62c" candidate_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4ef3d126d43e9be32731e708ff76c" is_pre_election: true
I20240501 14:04:29.722412 1323 raft_consensus.cc:2454] T 1074c96bc2564edca376f95bd195a62c P 05f4ef3d126d43e9be32731e708ff76c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d45680f41c84ac39bd97c6f00ac2b06 in term 0.
I20240501 14:04:29.722782 1128 leader_election.cc:302] T 1074c96bc2564edca376f95bd195a62c P 5d45680f41c84ac39bd97c6f00ac2b06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 05f4ef3d126d43e9be32731e708ff76c, 5d45680f41c84ac39bd97c6f00ac2b06; no voters:
I20240501 14:04:29.723058 2262 raft_consensus.cc:2781] T 1074c96bc2564edca376f95bd195a62c P 5d45680f41c84ac39bd97c6f00ac2b06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:04:29.723143 2262 raft_consensus.cc:482] T 1074c96bc2564edca376f95bd195a62c P 5d45680f41c84ac39bd97c6f00ac2b06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:04:29.723232 2262 raft_consensus.cc:3037] T 1074c96bc2564edca376f95bd195a62c P 5d45680f41c84ac39bd97c6f00ac2b06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:29.724967 2262 raft_consensus.cc:504] T 1074c96bc2564edca376f95bd195a62c P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } }
I20240501 14:04:29.725401 2262 leader_election.cc:288] T 1074c96bc2564edca376f95bd195a62c P 5d45680f41c84ac39bd97c6f00ac2b06 [CANDIDATE]: Term 1 election: Requested vote from peers 05f4ef3d126d43e9be32731e708ff76c (127.29.124.130:41511)
I20240501 14:04:29.725849 1323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1074c96bc2564edca376f95bd195a62c" candidate_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4ef3d126d43e9be32731e708ff76c"
I20240501 14:04:29.726029 1323 raft_consensus.cc:3037] T 1074c96bc2564edca376f95bd195a62c P 05f4ef3d126d43e9be32731e708ff76c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:29.727831 1323 raft_consensus.cc:2454] T 1074c96bc2564edca376f95bd195a62c P 05f4ef3d126d43e9be32731e708ff76c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d45680f41c84ac39bd97c6f00ac2b06 in term 1.
I20240501 14:04:29.728190 1128 leader_election.cc:302] T 1074c96bc2564edca376f95bd195a62c P 5d45680f41c84ac39bd97c6f00ac2b06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 05f4ef3d126d43e9be32731e708ff76c, 5d45680f41c84ac39bd97c6f00ac2b06; no voters:
I20240501 14:04:29.728427 2262 raft_consensus.cc:2781] T 1074c96bc2564edca376f95bd195a62c P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:04:29.728567 2262 raft_consensus.cc:686] T 1074c96bc2564edca376f95bd195a62c P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 LEADER]: Becoming Leader. State: Replica: 5d45680f41c84ac39bd97c6f00ac2b06, State: Running, Role: LEADER
I20240501 14:04:29.728798 2262 consensus_queue.cc:237] T 1074c96bc2564edca376f95bd195a62c P 5d45680f41c84ac39bd97c6f00ac2b06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } }
I20240501 14:04:29.731048 1052 catalog_manager.cc:5484] T 1074c96bc2564edca376f95bd195a62c P 5d45680f41c84ac39bd97c6f00ac2b06 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d45680f41c84ac39bd97c6f00ac2b06 (127.29.124.129). New cstate: current_term: 1 leader_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } health_report { overall_health: HEALTHY } } }
I20240501 14:04:29.970932 2253 consensus_queue.cc:1035] T 14e0e4808438484688605ff2d6d9258b P 05f4ef3d126d43e9be32731e708ff76c [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:04:30.059340 2285 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:04:30.059648 2285 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:04:30.066839 2285 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:04:30.096794 2262 consensus_queue.cc:1035] T 0f11914838ca4139a1c1813fc5334782 P 5d45680f41c84ac39bd97c6f00ac2b06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_3'
I20240501 14:04:30.197340 1323 raft_consensus.cc:1261] T 1074c96bc2564edca376f95bd195a62c P 05f4ef3d126d43e9be32731e708ff76c [term 1 FOLLOWER]: Refusing update from remote peer 5d45680f41c84ac39bd97c6f00ac2b06: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:04:30.197984 2262 consensus_queue.cc:1035] T 1074c96bc2564edca376f95bd195a62c P 5d45680f41c84ac39bd97c6f00ac2b06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 172.159 ms
time per row: 0.0860794 ms
I20240501 14:04:30.383209 1051 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50932:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 2
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:04:30.391492 1302 tablet_service.cc:1467] Processing CreateTablet for tablet db2b3f73d20c4790808bbe0f1d5e489c (DEFAULT_TABLE table=rebalance_test_table_4 [id=6fd9bb23d84f4078a694c2c7a0cb93b2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:30.391758 1172 tablet_service.cc:1467] Processing CreateTablet for tablet db2b3f73d20c4790808bbe0f1d5e489c (DEFAULT_TABLE table=rebalance_test_table_4 [id=6fd9bb23d84f4078a694c2c7a0cb93b2]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:30.391965 1302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db2b3f73d20c4790808bbe0f1d5e489c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:30.392172 1172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db2b3f73d20c4790808bbe0f1d5e489c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:30.392405 1173 tablet_service.cc:1467] Processing CreateTablet for tablet c586d994d2414073b25a8b3fab6a7046 (DEFAULT_TABLE table=rebalance_test_table_4 [id=6fd9bb23d84f4078a694c2c7a0cb93b2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:30.392678 1432 tablet_service.cc:1467] Processing CreateTablet for tablet 45d428dc499a4270b56a579f5b3cad83 (DEFAULT_TABLE table=rebalance_test_table_4 [id=6fd9bb23d84f4078a694c2c7a0cb93b2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:30.392796 1173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c586d994d2414073b25a8b3fab6a7046. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:30.393039 1432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45d428dc499a4270b56a579f5b3cad83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:30.398808 1433 tablet_service.cc:1467] Processing CreateTablet for tablet c586d994d2414073b25a8b3fab6a7046 (DEFAULT_TABLE table=rebalance_test_table_4 [id=6fd9bb23d84f4078a694c2c7a0cb93b2]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:30.399204 1433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c586d994d2414073b25a8b3fab6a7046. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:30.399554 1303 tablet_service.cc:1467] Processing CreateTablet for tablet 45d428dc499a4270b56a579f5b3cad83 (DEFAULT_TABLE table=rebalance_test_table_4 [id=6fd9bb23d84f4078a694c2c7a0cb93b2]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:04:30.399896 1303 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45d428dc499a4270b56a579f5b3cad83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:30.399917 2339 tablet_bootstrap.cc:492] T db2b3f73d20c4790808bbe0f1d5e489c P 5d45680f41c84ac39bd97c6f00ac2b06: Bootstrap starting.
I20240501 14:04:30.401707 2339 tablet_bootstrap.cc:654] T db2b3f73d20c4790808bbe0f1d5e489c P 5d45680f41c84ac39bd97c6f00ac2b06: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:30.406780 2338 tablet_bootstrap.cc:492] T db2b3f73d20c4790808bbe0f1d5e489c P 05f4ef3d126d43e9be32731e708ff76c: Bootstrap starting.
I20240501 14:04:30.406944 2339 tablet_bootstrap.cc:492] T db2b3f73d20c4790808bbe0f1d5e489c P 5d45680f41c84ac39bd97c6f00ac2b06: No bootstrap required, opened a new log
I20240501 14:04:30.407102 2339 ts_tablet_manager.cc:1367] T db2b3f73d20c4790808bbe0f1d5e489c P 5d45680f41c84ac39bd97c6f00ac2b06: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20240501 14:04:30.407696 2339 raft_consensus.cc:348] T db2b3f73d20c4790808bbe0f1d5e489c P 5d45680f41c84ac39bd97c6f00ac2b06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } }
I20240501 14:04:30.407860 2339 raft_consensus.cc:374] T db2b3f73d20c4790808bbe0f1d5e489c P 5d45680f41c84ac39bd97c6f00ac2b06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:30.407948 2339 raft_consensus.cc:729] T db2b3f73d20c4790808bbe0f1d5e489c P 5d45680f41c84ac39bd97c6f00ac2b06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d45680f41c84ac39bd97c6f00ac2b06, State: Initialized, Role: FOLLOWER
I20240501 14:04:30.408104 2338 tablet_bootstrap.cc:654] T db2b3f73d20c4790808bbe0f1d5e489c P 05f4ef3d126d43e9be32731e708ff76c: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:30.408156 2339 consensus_queue.cc:260] T db2b3f73d20c4790808bbe0f1d5e489c P 5d45680f41c84ac39bd97c6f00ac2b06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } }
I20240501 14:04:30.409380 2339 ts_tablet_manager.cc:1398] T db2b3f73d20c4790808bbe0f1d5e489c P 5d45680f41c84ac39bd97c6f00ac2b06: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:04:30.409545 2340 tablet_bootstrap.cc:492] T 45d428dc499a4270b56a579f5b3cad83 P eb004c2ac9d244f89b61448584d4115b: Bootstrap starting.
I20240501 14:04:30.409667 2339 tablet_bootstrap.cc:492] T c586d994d2414073b25a8b3fab6a7046 P 5d45680f41c84ac39bd97c6f00ac2b06: Bootstrap starting.
I20240501 14:04:30.410123 2338 tablet_bootstrap.cc:492] T db2b3f73d20c4790808bbe0f1d5e489c P 05f4ef3d126d43e9be32731e708ff76c: No bootstrap required, opened a new log
I20240501 14:04:30.410283 2338 ts_tablet_manager.cc:1367] T db2b3f73d20c4790808bbe0f1d5e489c P 05f4ef3d126d43e9be32731e708ff76c: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:04:30.411124 2340 tablet_bootstrap.cc:654] T 45d428dc499a4270b56a579f5b3cad83 P eb004c2ac9d244f89b61448584d4115b: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:30.411051 2338 raft_consensus.cc:348] T db2b3f73d20c4790808bbe0f1d5e489c P 05f4ef3d126d43e9be32731e708ff76c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } }
I20240501 14:04:30.411301 2338 raft_consensus.cc:374] T db2b3f73d20c4790808bbe0f1d5e489c P 05f4ef3d126d43e9be32731e708ff76c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:30.411407 2338 raft_consensus.cc:729] T db2b3f73d20c4790808bbe0f1d5e489c P 05f4ef3d126d43e9be32731e708ff76c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4ef3d126d43e9be32731e708ff76c, State: Initialized, Role: FOLLOWER
I20240501 14:04:30.411778 2339 tablet_bootstrap.cc:654] T c586d994d2414073b25a8b3fab6a7046 P 5d45680f41c84ac39bd97c6f00ac2b06: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:30.411685 2338 consensus_queue.cc:260] T db2b3f73d20c4790808bbe0f1d5e489c P 05f4ef3d126d43e9be32731e708ff76c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } }
I20240501 14:04:30.413132 2338 ts_tablet_manager.cc:1398] T db2b3f73d20c4790808bbe0f1d5e489c P 05f4ef3d126d43e9be32731e708ff76c: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:04:30.413794 2338 tablet_bootstrap.cc:492] T 45d428dc499a4270b56a579f5b3cad83 P 05f4ef3d126d43e9be32731e708ff76c: Bootstrap starting.
I20240501 14:04:30.415920 2338 tablet_bootstrap.cc:654] T 45d428dc499a4270b56a579f5b3cad83 P 05f4ef3d126d43e9be32731e708ff76c: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:30.416819 2340 tablet_bootstrap.cc:492] T 45d428dc499a4270b56a579f5b3cad83 P eb004c2ac9d244f89b61448584d4115b: No bootstrap required, opened a new log
I20240501 14:04:30.416958 2340 ts_tablet_manager.cc:1367] T 45d428dc499a4270b56a579f5b3cad83 P eb004c2ac9d244f89b61448584d4115b: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20240501 14:04:30.417205 2339 tablet_bootstrap.cc:492] T c586d994d2414073b25a8b3fab6a7046 P 5d45680f41c84ac39bd97c6f00ac2b06: No bootstrap required, opened a new log
I20240501 14:04:30.417378 2339 ts_tablet_manager.cc:1367] T c586d994d2414073b25a8b3fab6a7046 P 5d45680f41c84ac39bd97c6f00ac2b06: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20240501 14:04:30.417719 2340 raft_consensus.cc:348] T 45d428dc499a4270b56a579f5b3cad83 P eb004c2ac9d244f89b61448584d4115b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } }
I20240501 14:04:30.417894 2338 tablet_bootstrap.cc:492] T 45d428dc499a4270b56a579f5b3cad83 P 05f4ef3d126d43e9be32731e708ff76c: No bootstrap required, opened a new log
I20240501 14:04:30.417930 2340 raft_consensus.cc:374] T 45d428dc499a4270b56a579f5b3cad83 P eb004c2ac9d244f89b61448584d4115b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:30.418041 2338 ts_tablet_manager.cc:1367] T 45d428dc499a4270b56a579f5b3cad83 P 05f4ef3d126d43e9be32731e708ff76c: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:04:30.418054 2340 raft_consensus.cc:729] T 45d428dc499a4270b56a579f5b3cad83 P eb004c2ac9d244f89b61448584d4115b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb004c2ac9d244f89b61448584d4115b, State: Initialized, Role: FOLLOWER
I20240501 14:04:30.418179 2339 raft_consensus.cc:348] T c586d994d2414073b25a8b3fab6a7046 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } }
I20240501 14:04:30.418386 2339 raft_consensus.cc:374] T c586d994d2414073b25a8b3fab6a7046 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:30.418289 2340 consensus_queue.cc:260] T 45d428dc499a4270b56a579f5b3cad83 P eb004c2ac9d244f89b61448584d4115b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } }
I20240501 14:04:30.418462 2339 raft_consensus.cc:729] T c586d994d2414073b25a8b3fab6a7046 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d45680f41c84ac39bd97c6f00ac2b06, State: Initialized, Role: FOLLOWER
I20240501 14:04:30.418679 2339 consensus_queue.cc:260] T c586d994d2414073b25a8b3fab6a7046 P 5d45680f41c84ac39bd97c6f00ac2b06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } }
I20240501 14:04:30.418723 2338 raft_consensus.cc:348] T 45d428dc499a4270b56a579f5b3cad83 P 05f4ef3d126d43e9be32731e708ff76c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } }
I20240501 14:04:30.418884 2338 raft_consensus.cc:374] T 45d428dc499a4270b56a579f5b3cad83 P 05f4ef3d126d43e9be32731e708ff76c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:30.418954 2338 raft_consensus.cc:729] T 45d428dc499a4270b56a579f5b3cad83 P 05f4ef3d126d43e9be32731e708ff76c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 05f4ef3d126d43e9be32731e708ff76c, State: Initialized, Role: FOLLOWER
I20240501 14:04:30.419212 2338 consensus_queue.cc:260] T 45d428dc499a4270b56a579f5b3cad83 P 05f4ef3d126d43e9be32731e708ff76c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } }
I20240501 14:04:30.419754 2338 ts_tablet_manager.cc:1398] T 45d428dc499a4270b56a579f5b3cad83 P 05f4ef3d126d43e9be32731e708ff76c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:04:30.419823 2339 ts_tablet_manager.cc:1398] T c586d994d2414073b25a8b3fab6a7046 P 5d45680f41c84ac39bd97c6f00ac2b06: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:04:30.420099 2340 ts_tablet_manager.cc:1398] T 45d428dc499a4270b56a579f5b3cad83 P eb004c2ac9d244f89b61448584d4115b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:04:30.420306 2340 tablet_bootstrap.cc:492] T c586d994d2414073b25a8b3fab6a7046 P eb004c2ac9d244f89b61448584d4115b: Bootstrap starting.
I20240501 14:04:30.421435 2340 tablet_bootstrap.cc:654] T c586d994d2414073b25a8b3fab6a7046 P eb004c2ac9d244f89b61448584d4115b: Neither blocks nor log segments found. Creating new log.
I20240501 14:04:30.424852 2340 tablet_bootstrap.cc:492] T c586d994d2414073b25a8b3fab6a7046 P eb004c2ac9d244f89b61448584d4115b: No bootstrap required, opened a new log
I20240501 14:04:30.425016 2340 ts_tablet_manager.cc:1367] T c586d994d2414073b25a8b3fab6a7046 P eb004c2ac9d244f89b61448584d4115b: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.002s
I20240501 14:04:30.425671 2340 raft_consensus.cc:348] T c586d994d2414073b25a8b3fab6a7046 P eb004c2ac9d244f89b61448584d4115b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } }
I20240501 14:04:30.425825 2340 raft_consensus.cc:374] T c586d994d2414073b25a8b3fab6a7046 P eb004c2ac9d244f89b61448584d4115b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:04:30.425879 2340 raft_consensus.cc:729] T c586d994d2414073b25a8b3fab6a7046 P eb004c2ac9d244f89b61448584d4115b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb004c2ac9d244f89b61448584d4115b, State: Initialized, Role: FOLLOWER
I20240501 14:04:30.426039 2340 consensus_queue.cc:260] T c586d994d2414073b25a8b3fab6a7046 P eb004c2ac9d244f89b61448584d4115b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } }
I20240501 14:04:30.426415 2340 ts_tablet_manager.cc:1398] T c586d994d2414073b25a8b3fab6a7046 P eb004c2ac9d244f89b61448584d4115b: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:04:30.500221 2253 raft_consensus.cc:482] T db2b3f73d20c4790808bbe0f1d5e489c P 05f4ef3d126d43e9be32731e708ff76c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:04:30.500419 2253 raft_consensus.cc:504] T db2b3f73d20c4790808bbe0f1d5e489c P 05f4ef3d126d43e9be32731e708ff76c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } }
I20240501 14:04:30.500974 2253 leader_election.cc:288] T db2b3f73d20c4790808bbe0f1d5e489c P 05f4ef3d126d43e9be32731e708ff76c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d45680f41c84ac39bd97c6f00ac2b06 (127.29.124.129:40925)
I20240501 14:04:30.501539 1193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db2b3f73d20c4790808bbe0f1d5e489c" candidate_uuid: "05f4ef3d126d43e9be32731e708ff76c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" is_pre_election: true
I20240501 14:04:30.501755 1193 raft_consensus.cc:2454] T db2b3f73d20c4790808bbe0f1d5e489c P 5d45680f41c84ac39bd97c6f00ac2b06 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 05f4ef3d126d43e9be32731e708ff76c in term 0.
I20240501 14:04:30.502228 1256 leader_election.cc:302] T db2b3f73d20c4790808bbe0f1d5e489c P 05f4ef3d126d43e9be32731e708ff76c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 05f4ef3d126d43e9be32731e708ff76c, 5d45680f41c84ac39bd97c6f00ac2b06; no voters:
I20240501 14:04:30.502528 2253 raft_consensus.cc:2781] T db2b3f73d20c4790808bbe0f1d5e489c P 05f4ef3d126d43e9be32731e708ff76c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:04:30.502661 2253 raft_consensus.cc:482] T db2b3f73d20c4790808bbe0f1d5e489c P 05f4ef3d126d43e9be32731e708ff76c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:04:30.502748 2253 raft_consensus.cc:3037] T db2b3f73d20c4790808bbe0f1d5e489c P 05f4ef3d126d43e9be32731e708ff76c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:30.504582 2253 raft_consensus.cc:504] T db2b3f73d20c4790808bbe0f1d5e489c P 05f4ef3d126d43e9be32731e708ff76c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } }
I20240501 14:04:30.505017 2253 leader_election.cc:288] T db2b3f73d20c4790808bbe0f1d5e489c P 05f4ef3d126d43e9be32731e708ff76c [CANDIDATE]: Term 1 election: Requested vote from peers 5d45680f41c84ac39bd97c6f00ac2b06 (127.29.124.129:40925)
I20240501 14:04:30.505599 1193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db2b3f73d20c4790808bbe0f1d5e489c" candidate_uuid: "05f4ef3d126d43e9be32731e708ff76c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d45680f41c84ac39bd97c6f00ac2b06"
I20240501 14:04:30.505829 1193 raft_consensus.cc:3037] T db2b3f73d20c4790808bbe0f1d5e489c P 5d45680f41c84ac39bd97c6f00ac2b06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:30.508723 1193 raft_consensus.cc:2454] T db2b3f73d20c4790808bbe0f1d5e489c P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05f4ef3d126d43e9be32731e708ff76c in term 1.
I20240501 14:04:30.509106 1256 leader_election.cc:302] T db2b3f73d20c4790808bbe0f1d5e489c P 05f4ef3d126d43e9be32731e708ff76c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 05f4ef3d126d43e9be32731e708ff76c, 5d45680f41c84ac39bd97c6f00ac2b06; no voters:
I20240501 14:04:30.509418 2253 raft_consensus.cc:2781] T db2b3f73d20c4790808bbe0f1d5e489c P 05f4ef3d126d43e9be32731e708ff76c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:04:30.509548 2253 raft_consensus.cc:686] T db2b3f73d20c4790808bbe0f1d5e489c P 05f4ef3d126d43e9be32731e708ff76c [term 1 LEADER]: Becoming Leader. State: Replica: 05f4ef3d126d43e9be32731e708ff76c, State: Running, Role: LEADER
I20240501 14:04:30.509802 2253 consensus_queue.cc:237] T db2b3f73d20c4790808bbe0f1d5e489c P 05f4ef3d126d43e9be32731e708ff76c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } }
I20240501 14:04:30.512763 1051 catalog_manager.cc:5484] T db2b3f73d20c4790808bbe0f1d5e489c P 05f4ef3d126d43e9be32731e708ff76c reported cstate change: term changed from 0 to 1, leader changed from <none> to 05f4ef3d126d43e9be32731e708ff76c (127.29.124.130). New cstate: current_term: 1 leader_uuid: "05f4ef3d126d43e9be32731e708ff76c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } health_report { overall_health: UNKNOWN } } }
I20240501 14:04:30.751279 2262 raft_consensus.cc:482] T c586d994d2414073b25a8b3fab6a7046 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:04:30.751480 2262 raft_consensus.cc:504] T c586d994d2414073b25a8b3fab6a7046 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } }
I20240501 14:04:30.751933 2262 leader_election.cc:288] T c586d994d2414073b25a8b3fab6a7046 P 5d45680f41c84ac39bd97c6f00ac2b06 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eb004c2ac9d244f89b61448584d4115b (127.29.124.131:40225)
I20240501 14:04:30.752352 1453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c586d994d2414073b25a8b3fab6a7046" candidate_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb004c2ac9d244f89b61448584d4115b" is_pre_election: true
I20240501 14:04:30.752558 1453 raft_consensus.cc:2454] T c586d994d2414073b25a8b3fab6a7046 P eb004c2ac9d244f89b61448584d4115b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d45680f41c84ac39bd97c6f00ac2b06 in term 0.
I20240501 14:04:30.752926 1129 leader_election.cc:302] T c586d994d2414073b25a8b3fab6a7046 P 5d45680f41c84ac39bd97c6f00ac2b06 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5d45680f41c84ac39bd97c6f00ac2b06, eb004c2ac9d244f89b61448584d4115b; no voters:
I20240501 14:04:30.752946 2178 raft_consensus.cc:482] T 45d428dc499a4270b56a579f5b3cad83 P eb004c2ac9d244f89b61448584d4115b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:04:30.753060 2178 raft_consensus.cc:504] T 45d428dc499a4270b56a579f5b3cad83 P eb004c2ac9d244f89b61448584d4115b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } }
I20240501 14:04:30.753190 2262 raft_consensus.cc:2781] T c586d994d2414073b25a8b3fab6a7046 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:04:30.753325 2262 raft_consensus.cc:482] T c586d994d2414073b25a8b3fab6a7046 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:04:30.753407 2262 raft_consensus.cc:3037] T c586d994d2414073b25a8b3fab6a7046 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:30.753548 2178 leader_election.cc:288] T 45d428dc499a4270b56a579f5b3cad83 P eb004c2ac9d244f89b61448584d4115b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 05f4ef3d126d43e9be32731e708ff76c (127.29.124.130:41511)
I20240501 14:04:30.754046 1322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45d428dc499a4270b56a579f5b3cad83" candidate_uuid: "eb004c2ac9d244f89b61448584d4115b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4ef3d126d43e9be32731e708ff76c" is_pre_election: true
I20240501 14:04:30.754235 1322 raft_consensus.cc:2454] T 45d428dc499a4270b56a579f5b3cad83 P 05f4ef3d126d43e9be32731e708ff76c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb004c2ac9d244f89b61448584d4115b in term 0.
I20240501 14:04:30.754603 1388 leader_election.cc:302] T 45d428dc499a4270b56a579f5b3cad83 P eb004c2ac9d244f89b61448584d4115b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 05f4ef3d126d43e9be32731e708ff76c, eb004c2ac9d244f89b61448584d4115b; no voters:
I20240501 14:04:30.754834 2178 raft_consensus.cc:2781] T 45d428dc499a4270b56a579f5b3cad83 P eb004c2ac9d244f89b61448584d4115b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:04:30.754940 2178 raft_consensus.cc:482] T 45d428dc499a4270b56a579f5b3cad83 P eb004c2ac9d244f89b61448584d4115b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:04:30.755021 2178 raft_consensus.cc:3037] T 45d428dc499a4270b56a579f5b3cad83 P eb004c2ac9d244f89b61448584d4115b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:30.755546 2262 raft_consensus.cc:504] T c586d994d2414073b25a8b3fab6a7046 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } }
I20240501 14:04:30.755939 2262 leader_election.cc:288] T c586d994d2414073b25a8b3fab6a7046 P 5d45680f41c84ac39bd97c6f00ac2b06 [CANDIDATE]: Term 1 election: Requested vote from peers eb004c2ac9d244f89b61448584d4115b (127.29.124.131:40225)
I20240501 14:04:30.756430 1453 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c586d994d2414073b25a8b3fab6a7046" candidate_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eb004c2ac9d244f89b61448584d4115b"
I20240501 14:04:30.756620 1453 raft_consensus.cc:3037] T c586d994d2414073b25a8b3fab6a7046 P eb004c2ac9d244f89b61448584d4115b [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:30.757122 2178 raft_consensus.cc:504] T 45d428dc499a4270b56a579f5b3cad83 P eb004c2ac9d244f89b61448584d4115b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } }
I20240501 14:04:30.757612 2178 leader_election.cc:288] T 45d428dc499a4270b56a579f5b3cad83 P eb004c2ac9d244f89b61448584d4115b [CANDIDATE]: Term 1 election: Requested vote from peers 05f4ef3d126d43e9be32731e708ff76c (127.29.124.130:41511)
I20240501 14:04:30.758066 1322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "45d428dc499a4270b56a579f5b3cad83" candidate_uuid: "eb004c2ac9d244f89b61448584d4115b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "05f4ef3d126d43e9be32731e708ff76c"
I20240501 14:04:30.758248 1322 raft_consensus.cc:3037] T 45d428dc499a4270b56a579f5b3cad83 P 05f4ef3d126d43e9be32731e708ff76c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:04:30.759107 1453 raft_consensus.cc:2454] T c586d994d2414073b25a8b3fab6a7046 P eb004c2ac9d244f89b61448584d4115b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d45680f41c84ac39bd97c6f00ac2b06 in term 1.
I20240501 14:04:30.759462 1129 leader_election.cc:302] T c586d994d2414073b25a8b3fab6a7046 P 5d45680f41c84ac39bd97c6f00ac2b06 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5d45680f41c84ac39bd97c6f00ac2b06, eb004c2ac9d244f89b61448584d4115b; no voters:
I20240501 14:04:30.759697 2262 raft_consensus.cc:2781] T c586d994d2414073b25a8b3fab6a7046 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:04:30.759830 2262 raft_consensus.cc:686] T c586d994d2414073b25a8b3fab6a7046 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 LEADER]: Becoming Leader. State: Replica: 5d45680f41c84ac39bd97c6f00ac2b06, State: Running, Role: LEADER
I20240501 14:04:30.760174 2262 consensus_queue.cc:237] T c586d994d2414073b25a8b3fab6a7046 P 5d45680f41c84ac39bd97c6f00ac2b06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } }
I20240501 14:04:30.760566 1322 raft_consensus.cc:2454] T 45d428dc499a4270b56a579f5b3cad83 P 05f4ef3d126d43e9be32731e708ff76c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb004c2ac9d244f89b61448584d4115b in term 1.
I20240501 14:04:30.761396 1388 leader_election.cc:302] T 45d428dc499a4270b56a579f5b3cad83 P eb004c2ac9d244f89b61448584d4115b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 05f4ef3d126d43e9be32731e708ff76c, eb004c2ac9d244f89b61448584d4115b; no voters:
I20240501 14:04:30.761693 2178 raft_consensus.cc:2781] T 45d428dc499a4270b56a579f5b3cad83 P eb004c2ac9d244f89b61448584d4115b [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:04:30.762670 2178 raft_consensus.cc:686] T 45d428dc499a4270b56a579f5b3cad83 P eb004c2ac9d244f89b61448584d4115b [term 1 LEADER]: Becoming Leader. State: Replica: eb004c2ac9d244f89b61448584d4115b, State: Running, Role: LEADER
I20240501 14:04:30.762979 2178 consensus_queue.cc:237] T 45d428dc499a4270b56a579f5b3cad83 P eb004c2ac9d244f89b61448584d4115b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } }
I20240501 14:04:30.763440 1051 catalog_manager.cc:5484] T c586d994d2414073b25a8b3fab6a7046 P 5d45680f41c84ac39bd97c6f00ac2b06 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d45680f41c84ac39bd97c6f00ac2b06 (127.29.124.129). New cstate: current_term: 1 leader_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } health_report { overall_health: UNKNOWN } } }
I20240501 14:04:30.765350 1045 catalog_manager.cc:5484] T 45d428dc499a4270b56a579f5b3cad83 P eb004c2ac9d244f89b61448584d4115b reported cstate change: term changed from 0 to 1, leader changed from <none> to eb004c2ac9d244f89b61448584d4115b (127.29.124.131). New cstate: current_term: 1 leader_uuid: "eb004c2ac9d244f89b61448584d4115b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } health_report { overall_health: UNKNOWN } } }
I20240501 14:04:30.908442 2253 consensus_queue.cc:1035] T db2b3f73d20c4790808bbe0f1d5e489c P 05f4ef3d126d43e9be32731e708ff76c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:04:31.168140 2351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:04:31.168465 2351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:04:31.175904 2351 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240501 14:04:31.234751 2178 consensus_queue.cc:1035] T 45d428dc499a4270b56a579f5b3cad83 P eb004c2ac9d244f89b61448584d4115b [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:04:31.296207 1453 raft_consensus.cc:1261] T c586d994d2414073b25a8b3fab6a7046 P eb004c2ac9d244f89b61448584d4115b [term 1 FOLLOWER]: Refusing update from remote peer 5d45680f41c84ac39bd97c6f00ac2b06: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:04:31.297088 2262 consensus_queue.cc:1035] T c586d994d2414073b25a8b3fab6a7046 P 5d45680f41c84ac39bd97c6f00ac2b06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:04:31.320889 2391 mvcc.cc:205] Tried to move back new op lower bound from 7022888023222181888 to 7022888021035339776. Current Snapshot: MvccSnapshot[applied={T|T < 7022888023222181888}]
INSERT report
rows total: 2000
time total: 163.917 ms
time per row: 0.0819585 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240501 14:04:32.324798 30194 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUv7aek/build/asan/bin/kudu
/tmp/dist-test-taskUv7aek/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.124.132:34305
--local_ip_for_outbound_sockets=127.29.124.132
--tserver_master_addrs=127.29.124.190:46779
--webserver_port=42755
--webserver_interface=127.29.124.132
--builtin_ntp_servers=127.29.124.148:36129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 14:04:32.639302 2407 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:04:32.639590 2407 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:04:32.639705 2407 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:04:32.647085 2407 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:04:32.647220 2407 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:04:32.647364 2407 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.124.132
I20240501 14:04:32.656531 2407 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.124.148:36129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.124.132:34305
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.29.124.132
--webserver_port=42755
--tserver_master_addrs=127.29.124.190:46779
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.124.132
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:04:32.657809 2407 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:04:32.659449 2407 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:04:32.670811 2412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:04:32.671144 2413 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:04:32.671722 2415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:04:32.672003 2407 server_base.cc:1034] running on GCE node
I20240501 14:04:32.672631 2407 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:04:32.673554 2407 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:04:32.674750 2407 hybrid_clock.cc:648] HybridClock initialized: now 1714572272674684 us; error 60 us; skew 500 ppm
I20240501 14:04:32.675050 2407 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:04:32.680249 2407 webserver.cc:460] Webserver started at http://127.29.124.132:42755/ using document root <none> and password file <none>
I20240501 14:04:32.680919 2407 fs_manager.cc:359] Metadata directory not provided
I20240501 14:04:32.681037 2407 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:04:32.685834 2407 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:04:32.688334 2421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:04:32.689605 2407 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:04:32.689738 2407 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "953c332688d14831b3bc7cfd895e06e9"
format_stamp: "Formatted at 2024-05-01 14:04:19 on dist-test-slave-p83k"
I20240501 14:04:32.690477 2407 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:04:32.720505 2407 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:04:32.721216 2407 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:04:32.721486 2407 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:04:32.722326 2407 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:04:32.723706 2407 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:04:32.723773 2407 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:04:32.723889 2407 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:04:32.723937 2407 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:04:32.752014 2407 rpc_server.cc:307] RPC server started. Bound to: 127.29.124.132:34305
I20240501 14:04:32.752071 2533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.124.132:34305 every 8 connection(s)
I20240501 14:04:32.753371 2407 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 14:04:32.754243 30194 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUv7aek/build/asan/bin/kudu as pid 2407
I20240501 14:04:32.754870 30194 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUv7aek/build/asan/bin/kudu
/tmp/dist-test-taskUv7aek/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.124.133:34675
--local_ip_for_outbound_sockets=127.29.124.133
--tserver_master_addrs=127.29.124.190:46779
--webserver_port=46115
--webserver_interface=127.29.124.133
--builtin_ntp_servers=127.29.124.148:36129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:04:32.766049 2534 heartbeater.cc:347] Connected to a master server at 127.29.124.190:46779
I20240501 14:04:32.766352 2534 heartbeater.cc:464] Registering TS with master...
I20240501 14:04:32.767277 2534 heartbeater.cc:511] Master 127.29.124.190:46779 requested a full tablet report, sending...
I20240501 14:04:32.768781 1043 ts_manager.cc:194] Re-registered known tserver with Master: 953c332688d14831b3bc7cfd895e06e9 (127.29.124.132:34305)
I20240501 14:04:32.769649 1043 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.124.132:45253
W20240501 14:04:33.068912 2538 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:04:33.069169 2538 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:04:33.069312 2538 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:04:33.076573 2538 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:04:33.076686 2538 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:04:33.076828 2538 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.124.133
I20240501 14:04:33.085363 2538 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.124.148:36129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.124.133:34675
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.29.124.133
--webserver_port=46115
--tserver_master_addrs=127.29.124.190:46779
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.124.133
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:04:33.086560 2538 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:04:33.088191 2538 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:04:33.099167 2543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:04:33.099581 2546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:04:33.099915 2544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:04:33.100383 2538 server_base.cc:1034] running on GCE node
I20240501 14:04:33.100893 2538 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:04:33.101800 2538 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:04:33.102988 2538 hybrid_clock.cc:648] HybridClock initialized: now 1714572273102951 us; error 28 us; skew 500 ppm
I20240501 14:04:33.103277 2538 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:04:33.105381 2538 webserver.cc:460] Webserver started at http://127.29.124.133:46115/ using document root <none> and password file <none>
I20240501 14:04:33.106092 2538 fs_manager.cc:359] Metadata directory not provided
I20240501 14:04:33.106207 2538 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:04:33.111515 2538 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.003s
I20240501 14:04:33.114550 2552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:04:33.115815 2538 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:04:33.115960 2538 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "4e6923f9753a4450ac4343c34c427d1e"
format_stamp: "Formatted at 2024-05-01 14:04:20 on dist-test-slave-p83k"
I20240501 14:04:33.116765 2538 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:04:33.129078 2538 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:04:33.129803 2538 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:04:33.130050 2538 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:04:33.130865 2538 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:04:33.132575 2559 ts_tablet_manager.cc:522] Loading tablet metadata (0/1 complete)
I20240501 14:04:33.137924 2538 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240501 14:04:33.138029 2538 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20240501 14:04:33.138159 2538 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240501 14:04:33.141014 2538 ts_tablet_manager.cc:590] Registered 1 tablets
I20240501 14:04:33.141104 2538 ts_tablet_manager.cc:569] Time spent register tablets: real 0.003s user 0.000s sys 0.000s
I20240501 14:04:33.141444 2559 tablet_bootstrap.cc:492] T 4da90afe36fd411c8bfd661e4881f86c P 4e6923f9753a4450ac4343c34c427d1e: Bootstrap starting.
I20240501 14:04:33.156774 2559 log.cc:824] T 4da90afe36fd411c8bfd661e4881f86c P 4e6923f9753a4450ac4343c34c427d1e: Log is configured to *not* fsync() on all Append() calls
I20240501 14:04:33.166169 2559 tablet_bootstrap.cc:492] T 4da90afe36fd411c8bfd661e4881f86c P 4e6923f9753a4450ac4343c34c427d1e: Bootstrap replayed 1/1 log segments. Stats: ops{read=0 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:04:33.166895 2559 tablet_bootstrap.cc:492] T 4da90afe36fd411c8bfd661e4881f86c P 4e6923f9753a4450ac4343c34c427d1e: Bootstrap complete.
I20240501 14:04:33.168515 2559 ts_tablet_manager.cc:1367] T 4da90afe36fd411c8bfd661e4881f86c P 4e6923f9753a4450ac4343c34c427d1e: Time spent bootstrapping tablet: real 0.027s user 0.016s sys 0.008s
I20240501 14:04:33.180153 2538 rpc_server.cc:307] RPC server started. Bound to: 127.29.124.133:34675
I20240501 14:04:33.180208 2666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.124.133:34675 every 8 connection(s)
I20240501 14:04:33.181830 2538 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 14:04:33.181319 2559 raft_consensus.cc:348] T 4da90afe36fd411c8bfd661e4881f86c P 4e6923f9753a4450ac4343c34c427d1e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } }
I20240501 14:04:33.182356 2559 raft_consensus.cc:729] T 4da90afe36fd411c8bfd661e4881f86c P 4e6923f9753a4450ac4343c34c427d1e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e6923f9753a4450ac4343c34c427d1e, State: Initialized, Role: FOLLOWER
I20240501 14:04:33.183385 2559 consensus_queue.cc:260] T 4da90afe36fd411c8bfd661e4881f86c P 4e6923f9753a4450ac4343c34c427d1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } }
I20240501 14:04:33.184868 2559 ts_tablet_manager.cc:1398] T 4da90afe36fd411c8bfd661e4881f86c P 4e6923f9753a4450ac4343c34c427d1e: Time spent starting tablet: real 0.016s user 0.014s sys 0.004s
I20240501 14:04:33.186728 30194 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUv7aek/build/asan/bin/kudu as pid 2538
I20240501 14:04:33.187427 30194 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUv7aek/build/asan/bin/kudu
/tmp/dist-test-taskUv7aek/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.124.134:37211
--local_ip_for_outbound_sockets=127.29.124.134
--tserver_master_addrs=127.29.124.190:46779
--webserver_port=41379
--webserver_interface=127.29.124.134
--builtin_ntp_servers=127.29.124.148:36129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:04:33.194514 2667 heartbeater.cc:347] Connected to a master server at 127.29.124.190:46779
I20240501 14:04:33.194800 2667 heartbeater.cc:464] Registering TS with master...
I20240501 14:04:33.195569 2667 heartbeater.cc:511] Master 127.29.124.190:46779 requested a full tablet report, sending...
I20240501 14:04:33.197858 1043 ts_manager.cc:194] Re-registered known tserver with Master: 4e6923f9753a4450ac4343c34c427d1e (127.29.124.133:34675)
I20240501 14:04:33.199290 1043 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.124.133:38581
I20240501 14:04:33.202105 2667 heartbeater.cc:503] Master 127.29.124.190:46779 was elected leader, sending a full tablet report...
W20240501 14:04:33.433537 2668 tablet.cc:2369] T 4da90afe36fd411c8bfd661e4881f86c P 4e6923f9753a4450ac4343c34c427d1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 14:04:33.511052 2671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:04:33.511319 2671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:04:33.511425 2671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:04:33.518692 2671 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:04:33.518826 2671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:04:33.518975 2671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.124.134
I20240501 14:04:33.527729 2671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.124.148:36129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.124.134:37211
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.29.124.134
--webserver_port=41379
--tserver_master_addrs=127.29.124.190:46779
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.124.134
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:04:33.528957 2671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:04:33.530687 2671 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:04:33.541560 2676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:04:33.541945 2677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:04:33.542348 2679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:04:33.542574 2671 server_base.cc:1034] running on GCE node
I20240501 14:04:33.543112 2671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:04:33.543988 2671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:04:33.545183 2671 hybrid_clock.cc:648] HybridClock initialized: now 1714572273545140 us; error 39 us; skew 500 ppm
I20240501 14:04:33.545502 2671 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:04:33.547781 2671 webserver.cc:460] Webserver started at http://127.29.124.134:41379/ using document root <none> and password file <none>
I20240501 14:04:33.548686 2671 fs_manager.cc:359] Metadata directory not provided
I20240501 14:04:33.548795 2671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:04:33.553802 2671 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:04:33.556455 2685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:04:33.557850 2671 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:04:33.558005 2671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "32064197afde4b79bf388b080c365d88"
format_stamp: "Formatted at 2024-05-01 14:04:20 on dist-test-slave-p83k"
I20240501 14:04:33.558779 2671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:04:33.576205 2671 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:04:33.576983 2671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:04:33.577239 2671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:04:33.578202 2671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:04:33.579635 2671 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:04:33.579706 2671 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:04:33.579797 2671 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:04:33.579838 2671 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:04:33.607441 2671 rpc_server.cc:307] RPC server started. Bound to: 127.29.124.134:37211
I20240501 14:04:33.607476 2797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.124.134:37211 every 8 connection(s)
I20240501 14:04:33.608814 2671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 14:04:33.611025 30194 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUv7aek/build/asan/bin/kudu as pid 2671
I20240501 14:04:33.611768 30194 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUv7aek/build/asan/bin/kudu
/tmp/dist-test-taskUv7aek/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.124.135:42923
--local_ip_for_outbound_sockets=127.29.124.135
--tserver_master_addrs=127.29.124.190:46779
--webserver_port=41465
--webserver_interface=127.29.124.135
--builtin_ntp_servers=127.29.124.148:36129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:04:33.622071 2798 heartbeater.cc:347] Connected to a master server at 127.29.124.190:46779
I20240501 14:04:33.622368 2798 heartbeater.cc:464] Registering TS with master...
I20240501 14:04:33.623193 2798 heartbeater.cc:511] Master 127.29.124.190:46779 requested a full tablet report, sending...
I20240501 14:04:33.624651 1043 ts_manager.cc:194] Re-registered known tserver with Master: 32064197afde4b79bf388b080c365d88 (127.29.124.134:37211)
I20240501 14:04:33.625741 1043 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.124.134:36603
I20240501 14:04:33.771919 2534 heartbeater.cc:503] Master 127.29.124.190:46779 was elected leader, sending a full tablet report...
W20240501 14:04:33.932947 2801 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:04:33.933210 2801 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:04:33.933342 2801 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:04:33.940693 2801 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:04:33.940806 2801 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:04:33.940958 2801 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.124.135
I20240501 14:04:33.950258 2801 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.124.148:36129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.124.135:42923
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.29.124.135
--webserver_port=41465
--tserver_master_addrs=127.29.124.190:46779
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.124.135
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:04:33.951514 2801 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:04:33.953156 2801 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:04:33.964721 2806 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:04:33.965011 2807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:04:33.965392 2809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:04:33.965876 2801 server_base.cc:1034] running on GCE node
I20240501 14:04:33.966440 2801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:04:33.967357 2801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:04:33.968540 2801 hybrid_clock.cc:648] HybridClock initialized: now 1714572273968481 us; error 52 us; skew 500 ppm
I20240501 14:04:33.968829 2801 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:04:33.970979 2801 webserver.cc:460] Webserver started at http://127.29.124.135:41465/ using document root <none> and password file <none>
I20240501 14:04:33.971704 2801 fs_manager.cc:359] Metadata directory not provided
I20240501 14:04:33.971822 2801 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:04:33.977116 2801 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20240501 14:04:33.980064 2815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:04:33.981624 2801 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20240501 14:04:33.981779 2801 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "752bc4569da647679d5df51641961840"
format_stamp: "Formatted at 2024-05-01 14:04:21 on dist-test-slave-p83k"
I20240501 14:04:33.982901 2801 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:04:34.006693 2801 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:04:34.007516 2801 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:04:34.007756 2801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:04:34.008667 2801 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:04:34.010521 2822 ts_tablet_manager.cc:522] Loading tablet metadata (0/1 complete)
I20240501 14:04:34.015933 2801 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240501 14:04:34.016026 2801 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20240501 14:04:34.016152 2801 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240501 14:04:34.019073 2801 ts_tablet_manager.cc:590] Registered 1 tablets
I20240501 14:04:34.019166 2801 ts_tablet_manager.cc:569] Time spent register tablets: real 0.003s user 0.000s sys 0.001s
I20240501 14:04:34.019543 2822 tablet_bootstrap.cc:492] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840: Bootstrap starting.
I20240501 14:04:34.039089 2822 log.cc:824] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840: Log is configured to *not* fsync() on all Append() calls
I20240501 14:04:34.042906 2822 tablet_bootstrap.cc:492] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=0 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 14:04:34.043489 2822 tablet_bootstrap.cc:492] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840: Bootstrap complete.
I20240501 14:04:34.044416 2822 ts_tablet_manager.cc:1367] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840: Time spent bootstrapping tablet: real 0.025s user 0.012s sys 0.009s
I20240501 14:04:34.054313 2822 raft_consensus.cc:348] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840 [term 1 FOLLOWER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } }
I20240501 14:04:34.055949 2822 raft_consensus.cc:729] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 752bc4569da647679d5df51641961840, State: Initialized, Role: FOLLOWER
I20240501 14:04:34.056895 2822 consensus_queue.cc:260] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } }
I20240501 14:04:34.058951 2822 ts_tablet_manager.cc:1398] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840: Time spent starting tablet: real 0.014s user 0.013s sys 0.003s
I20240501 14:04:34.060098 2801 rpc_server.cc:307] RPC server started. Bound to: 127.29.124.135:42923
I20240501 14:04:34.060161 2930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.124.135:42923 every 8 connection(s)
I20240501 14:04:34.061288 2801 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/data/info.pb
W20240501 14:04:34.062386 2932 tablet.cc:2369] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:04:34.064584 30194 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUv7aek/build/asan/bin/kudu as pid 2801
I20240501 14:04:34.064898 30194 test_util.cc:268] Using random seed: -145334828
I20240501 14:04:34.073895 2931 heartbeater.cc:347] Connected to a master server at 127.29.124.190:46779
I20240501 14:04:34.074203 2931 heartbeater.cc:464] Registering TS with master...
I20240501 14:04:34.075135 2931 heartbeater.cc:511] Master 127.29.124.190:46779 requested a full tablet report, sending...
I20240501 14:04:34.077638 1043 ts_manager.cc:194] Re-registered known tserver with Master: 752bc4569da647679d5df51641961840 (127.29.124.135:42923)
I20240501 14:04:34.078811 1043 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.124.135:37755
I20240501 14:04:34.081521 30194 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_0 already exists
I20240501 14:04:34.081949 30194 test_util.cc:268] Using random seed: -145317767
I20240501 14:04:34.097478 30194 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_1 already exists
I20240501 14:04:34.108305 30194 test_util.cc:268] Using random seed: -145291436
I20240501 14:04:34.138988 30194 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_2 already exists
I20240501 14:04:34.149741 30194 test_util.cc:268] Using random seed: -145250009
I20240501 14:04:34.196673 30194 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_3 already exists
I20240501 14:04:34.203675 30194 test_util.cc:268] Using random seed: -145196062
I20240501 14:04:34.256981 30194 test_workload.cc:401] TestWorkload: Skipping table creation because table rebalance_test_table_4 already exists
I20240501 14:04:34.628159 2798 heartbeater.cc:503] Master 127.29.124.190:46779 was elected leader, sending a full tablet report...
I20240501 14:04:34.894997 3035 raft_consensus.cc:482] T 4da90afe36fd411c8bfd661e4881f86c P 4e6923f9753a4450ac4343c34c427d1e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:04:34.895478 3035 raft_consensus.cc:504] T 4da90afe36fd411c8bfd661e4881f86c P 4e6923f9753a4450ac4343c34c427d1e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } }
I20240501 14:04:34.910187 3035 leader_election.cc:288] T 4da90afe36fd411c8bfd661e4881f86c P 4e6923f9753a4450ac4343c34c427d1e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 752bc4569da647679d5df51641961840 (127.29.124.135:42923)
I20240501 14:04:34.980381 2885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4da90afe36fd411c8bfd661e4881f86c" candidate_uuid: "4e6923f9753a4450ac4343c34c427d1e" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "752bc4569da647679d5df51641961840" is_pre_election: true
I20240501 14:04:34.980890 2885 raft_consensus.cc:2396] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4e6923f9753a4450ac4343c34c427d1e for term 2 because replica has last-logged OpId of term: 1 index: 1, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20240501 14:04:34.983188 2555 leader_election.cc:302] T 4da90afe36fd411c8bfd661e4881f86c P 4e6923f9753a4450ac4343c34c427d1e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 4e6923f9753a4450ac4343c34c427d1e; no voters: 752bc4569da647679d5df51641961840
I20240501 14:04:34.983857 3035 raft_consensus.cc:2726] T 4da90afe36fd411c8bfd661e4881f86c P 4e6923f9753a4450ac4343c34c427d1e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240501 14:04:35.081431 2931 heartbeater.cc:503] Master 127.29.124.190:46779 was elected leader, sending a full tablet report...
I20240501 14:04:35.465997 1302 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:35.491293 1172 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:35.530129 2733 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:04:35.530278 2602 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:04:35.534622 1432 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:04:35.537676 2864 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:04:35.652717 2469 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:04:35.720999 3084 raft_consensus.cc:482] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:04:35.721397 3084 raft_consensus.cc:504] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } }
I20240501 14:04:35.739198 3084 leader_election.cc:288] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4e6923f9753a4450ac4343c34c427d1e (127.29.124.133:34675)
I20240501 14:04:35.760836 2622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4da90afe36fd411c8bfd661e4881f86c" candidate_uuid: "752bc4569da647679d5df51641961840" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4e6923f9753a4450ac4343c34c427d1e" is_pre_election: true
I20240501 14:04:35.761224 2622 raft_consensus.cc:2454] T 4da90afe36fd411c8bfd661e4881f86c P 4e6923f9753a4450ac4343c34c427d1e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 752bc4569da647679d5df51641961840 in term 1.
I20240501 14:04:35.762043 2816 leader_election.cc:302] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4e6923f9753a4450ac4343c34c427d1e, 752bc4569da647679d5df51641961840; no voters:
I20240501 14:04:35.762471 3084 raft_consensus.cc:2781] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240501 14:04:35.762620 3084 raft_consensus.cc:482] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:04:35.762713 3084 raft_consensus.cc:3037] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:04:35.770960 3084 raft_consensus.cc:504] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } }
I20240501 14:04:35.774992 3084 leader_election.cc:288] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840 [CANDIDATE]: Term 2 election: Requested vote from peers 4e6923f9753a4450ac4343c34c427d1e (127.29.124.133:34675)
I20240501 14:04:35.775583 2622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4da90afe36fd411c8bfd661e4881f86c" candidate_uuid: "752bc4569da647679d5df51641961840" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4e6923f9753a4450ac4343c34c427d1e"
I20240501 14:04:35.775785 2622 raft_consensus.cc:3037] T 4da90afe36fd411c8bfd661e4881f86c P 4e6923f9753a4450ac4343c34c427d1e [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:04:35.788861 2622 raft_consensus.cc:2454] T 4da90afe36fd411c8bfd661e4881f86c P 4e6923f9753a4450ac4343c34c427d1e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 752bc4569da647679d5df51641961840 in term 2.
I20240501 14:04:35.792301 2816 leader_election.cc:302] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4e6923f9753a4450ac4343c34c427d1e, 752bc4569da647679d5df51641961840; no voters:
I20240501 14:04:35.792958 3084 raft_consensus.cc:2781] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:04:35.806951 3084 raft_consensus.cc:686] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840 [term 2 LEADER]: Becoming Leader. State: Replica: 752bc4569da647679d5df51641961840, State: Running, Role: LEADER
I20240501 14:04:35.807525 3084 consensus_queue.cc:237] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } }
I20240501 14:04:35.819123 1045 catalog_manager.cc:5484] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "752bc4569da647679d5df51641961840" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } health_report { overall_health: HEALTHY } } }
I20240501 14:04:36.363435 2622 raft_consensus.cc:1261] T 4da90afe36fd411c8bfd661e4881f86c P 4e6923f9753a4450ac4343c34c427d1e [term 2 FOLLOWER]: Refusing update from remote peer 752bc4569da647679d5df51641961840: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240501 14:04:36.364285 3127 consensus_queue.cc:1035] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:04:36.383767 1302 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:36.476719 2864 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:36.490175 2733 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:04:36.543298 2469 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:04:36.580602 2602 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:04:36.651286 1172 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:36.674794 1432 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:04:36.870607 1192 consensus_queue.cc:237] T c586d994d2414073b25a8b3fab6a7046 P 5d45680f41c84ac39bd97c6f00ac2b06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 46, Committed index: 46, Last appended: 1.46, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 47 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: NON_VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: true } }
I20240501 14:04:36.874217 1452 raft_consensus.cc:1261] T c586d994d2414073b25a8b3fab6a7046 P eb004c2ac9d244f89b61448584d4115b [term 1 FOLLOWER]: Refusing update from remote peer 5d45680f41c84ac39bd97c6f00ac2b06: Log matching property violated. Preceding OpId in replica: term: 1 index: 46. Preceding OpId from leader: term: 1 index: 47. (index mismatch)
I20240501 14:04:36.876729 3105 consensus_queue.cc:1035] T c586d994d2414073b25a8b3fab6a7046 P 5d45680f41c84ac39bd97c6f00ac2b06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 47, Last known committed idx: 46, Time since last communication: 0.000s
I20240501 14:04:36.880033 3105 raft_consensus.cc:2932] T c586d994d2414073b25a8b3fab6a7046 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 LEADER]: Committing config change with OpId 1.47: config changed from index -1 to 47, NON_VOTER 953c332688d14831b3bc7cfd895e06e9 (127.29.124.132) added. New config: { opid_index: 47 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: NON_VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: true } } }
W20240501 14:04:36.885087 1128 consensus_peers.cc:485] T c586d994d2414073b25a8b3fab6a7046 P 5d45680f41c84ac39bd97c6f00ac2b06 -> Peer 953c332688d14831b3bc7cfd895e06e9 (127.29.124.132:34305): Couldn't send request to peer 953c332688d14831b3bc7cfd895e06e9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c586d994d2414073b25a8b3fab6a7046. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:04:36.894070 1045 catalog_manager.cc:5484] T c586d994d2414073b25a8b3fab6a7046 P 5d45680f41c84ac39bd97c6f00ac2b06 reported cstate change: config changed from index -1 to 47, NON_VOTER 953c332688d14831b3bc7cfd895e06e9 (127.29.124.132) added. New cstate: current_term: 1 leader_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" committed_config { opid_index: 47 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: NON_VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:04:36.894590 1452 raft_consensus.cc:2932] T c586d994d2414073b25a8b3fab6a7046 P eb004c2ac9d244f89b61448584d4115b [term 1 FOLLOWER]: Committing config change with OpId 1.47: config changed from index -1 to 47, NON_VOTER 953c332688d14831b3bc7cfd895e06e9 (127.29.124.132) added. New config: { opid_index: 47 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: NON_VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: true } } }
I20240501 14:04:37.023376 1192 consensus_queue.cc:237] T 1074c96bc2564edca376f95bd195a62c P 5d45680f41c84ac39bd97c6f00ac2b06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 61, Committed index: 61, Last appended: 1.62, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "32064197afde4b79bf388b080c365d88" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 37211 } attrs { promote: true } }
I20240501 14:04:37.025800 1323 raft_consensus.cc:1261] T 1074c96bc2564edca376f95bd195a62c P 05f4ef3d126d43e9be32731e708ff76c [term 1 FOLLOWER]: Refusing update from remote peer 5d45680f41c84ac39bd97c6f00ac2b06: Log matching property violated. Preceding OpId in replica: term: 1 index: 62. Preceding OpId from leader: term: 1 index: 63. (index mismatch)
I20240501 14:04:37.035673 3105 consensus_queue.cc:1035] T 1074c96bc2564edca376f95bd195a62c P 5d45680f41c84ac39bd97c6f00ac2b06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 63, Last known committed idx: 61, Time since last communication: 0.000s
W20240501 14:04:37.051630 1126 consensus_peers.cc:485] T 1074c96bc2564edca376f95bd195a62c P 5d45680f41c84ac39bd97c6f00ac2b06 -> Peer 32064197afde4b79bf388b080c365d88 (127.29.124.134:37211): Couldn't send request to peer 32064197afde4b79bf388b080c365d88. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1074c96bc2564edca376f95bd195a62c. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:04:37.054072 3105 raft_consensus.cc:2932] T 1074c96bc2564edca376f95bd195a62c P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 LEADER]: Committing config change with OpId 1.63: config changed from index -1 to 63, NON_VOTER 32064197afde4b79bf388b080c365d88 (127.29.124.134) added. New config: { opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "32064197afde4b79bf388b080c365d88" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 37211 } attrs { promote: true } } }
I20240501 14:04:37.058705 1050 catalog_manager.cc:5484] T 1074c96bc2564edca376f95bd195a62c P 5d45680f41c84ac39bd97c6f00ac2b06 reported cstate change: config changed from index -1 to 63, NON_VOTER 32064197afde4b79bf388b080c365d88 (127.29.124.134) added. New cstate: current_term: 1 leader_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" committed_config { opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32064197afde4b79bf388b080c365d88" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 37211 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:04:37.059312 1322 raft_consensus.cc:2932] T 1074c96bc2564edca376f95bd195a62c P 05f4ef3d126d43e9be32731e708ff76c [term 1 FOLLOWER]: Committing config change with OpId 1.63: config changed from index -1 to 63, NON_VOTER 32064197afde4b79bf388b080c365d88 (127.29.124.134) added. New config: { opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "32064197afde4b79bf388b080c365d88" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 37211 } attrs { promote: true } } }
I20240501 14:04:37.155066 1323 consensus_queue.cc:237] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 05f4ef3d126d43e9be32731e708ff76c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 69, Committed index: 69, Last appended: 1.69, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: true } }
I20240501 14:04:37.177917 1192 raft_consensus.cc:1261] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 FOLLOWER]: Refusing update from remote peer 05f4ef3d126d43e9be32731e708ff76c: Log matching property violated. Preceding OpId in replica: term: 1 index: 69. Preceding OpId from leader: term: 1 index: 70. (index mismatch)
I20240501 14:04:37.178475 3046 consensus_queue.cc:1035] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 05f4ef3d126d43e9be32731e708ff76c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 70, Last known committed idx: 69, Time since last communication: 0.000s
I20240501 14:04:37.182488 2989 raft_consensus.cc:2932] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 05f4ef3d126d43e9be32731e708ff76c [term 1 LEADER]: Committing config change with OpId 1.70: config changed from index -1 to 70, NON_VOTER 752bc4569da647679d5df51641961840 (127.29.124.135) added. New config: { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: true } } }
I20240501 14:04:37.187177 1192 raft_consensus.cc:2932] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 FOLLOWER]: Committing config change with OpId 1.70: config changed from index -1 to 70, NON_VOTER 752bc4569da647679d5df51641961840 (127.29.124.135) added. New config: { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: true } } }
I20240501 14:04:37.188102 1050 catalog_manager.cc:5484] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 05f4ef3d126d43e9be32731e708ff76c reported cstate change: config changed from index -1 to 70, NON_VOTER 752bc4569da647679d5df51641961840 (127.29.124.135) added. New cstate: current_term: 1 leader_uuid: "05f4ef3d126d43e9be32731e708ff76c" committed_config { opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 14:04:37.227013 1258 consensus_peers.cc:485] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 05f4ef3d126d43e9be32731e708ff76c -> Peer 752bc4569da647679d5df51641961840 (127.29.124.135:42923): Couldn't send request to peer 752bc4569da647679d5df51641961840. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3a4ca3f4edcd4bf29d0bab40a8856039. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:04:37.258319 1191 consensus_queue.cc:237] T 0f11914838ca4139a1c1813fc5334782 P 5d45680f41c84ac39bd97c6f00ac2b06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 48, Committed index: 48, Last appended: 1.49, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: true } }
W20240501 14:04:37.269495 1126 consensus_peers.cc:485] T 0f11914838ca4139a1c1813fc5334782 P 5d45680f41c84ac39bd97c6f00ac2b06 -> Peer 4e6923f9753a4450ac4343c34c427d1e (127.29.124.133:34675): Couldn't send request to peer 4e6923f9753a4450ac4343c34c427d1e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f11914838ca4139a1c1813fc5334782. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:04:37.271006 1451 raft_consensus.cc:1261] T 0f11914838ca4139a1c1813fc5334782 P eb004c2ac9d244f89b61448584d4115b [term 1 FOLLOWER]: Refusing update from remote peer 5d45680f41c84ac39bd97c6f00ac2b06: Log matching property violated. Preceding OpId in replica: term: 1 index: 48. Preceding OpId from leader: term: 1 index: 50. (index mismatch)
I20240501 14:04:37.273622 3153 consensus_queue.cc:1035] T 0f11914838ca4139a1c1813fc5334782 P 5d45680f41c84ac39bd97c6f00ac2b06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 50, Last known committed idx: 48, Time since last communication: 0.000s
I20240501 14:04:37.281306 1453 raft_consensus.cc:2932] T 0f11914838ca4139a1c1813fc5334782 P eb004c2ac9d244f89b61448584d4115b [term 1 FOLLOWER]: Committing config change with OpId 1.50: config changed from index -1 to 50, NON_VOTER 4e6923f9753a4450ac4343c34c427d1e (127.29.124.133) added. New config: { opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: true } } }
I20240501 14:04:37.282555 3153 raft_consensus.cc:2932] T 0f11914838ca4139a1c1813fc5334782 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 LEADER]: Committing config change with OpId 1.50: config changed from index -1 to 50, NON_VOTER 4e6923f9753a4450ac4343c34c427d1e (127.29.124.133) added. New config: { opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: true } } }
I20240501 14:04:37.286883 1045 catalog_manager.cc:5484] T 0f11914838ca4139a1c1813fc5334782 P eb004c2ac9d244f89b61448584d4115b reported cstate change: config changed from index -1 to 50, NON_VOTER 4e6923f9753a4450ac4343c34c427d1e (127.29.124.133) added. New cstate: current_term: 1 leader_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" committed_config { opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: true } } }
I20240501 14:04:37.345963 1192 consensus_queue.cc:237] T 8979fd4d21814622ba687edb88fa7851 P 5d45680f41c84ac39bd97c6f00ac2b06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 92, Committed index: 92, Last appended: 1.93, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: NON_VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: true } }
W20240501 14:04:37.349362 1128 consensus_peers.cc:485] T 8979fd4d21814622ba687edb88fa7851 P 5d45680f41c84ac39bd97c6f00ac2b06 -> Peer 953c332688d14831b3bc7cfd895e06e9 (127.29.124.132:34305): Couldn't send request to peer 953c332688d14831b3bc7cfd895e06e9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8979fd4d21814622ba687edb88fa7851. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:04:37.351748 1322 raft_consensus.cc:1261] T 8979fd4d21814622ba687edb88fa7851 P 05f4ef3d126d43e9be32731e708ff76c [term 1 FOLLOWER]: Refusing update from remote peer 5d45680f41c84ac39bd97c6f00ac2b06: Log matching property violated. Preceding OpId in replica: term: 1 index: 93. Preceding OpId from leader: term: 1 index: 94. (index mismatch)
I20240501 14:04:37.352859 3171 consensus_queue.cc:1035] T 8979fd4d21814622ba687edb88fa7851 P 5d45680f41c84ac39bd97c6f00ac2b06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 94, Last known committed idx: 92, Time since last communication: 0.001s
I20240501 14:04:37.357508 3105 raft_consensus.cc:2932] T 8979fd4d21814622ba687edb88fa7851 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 LEADER]: Committing config change with OpId 1.94: config changed from index -1 to 94, NON_VOTER 953c332688d14831b3bc7cfd895e06e9 (127.29.124.132) added. New config: { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: NON_VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: true } } }
I20240501 14:04:37.358952 3187 ts_tablet_manager.cc:902] T c586d994d2414073b25a8b3fab6a7046 P 953c332688d14831b3bc7cfd895e06e9: Initiating tablet copy from peer 5d45680f41c84ac39bd97c6f00ac2b06 (127.29.124.129:40925)
I20240501 14:04:37.358795 1323 raft_consensus.cc:2932] T 8979fd4d21814622ba687edb88fa7851 P 05f4ef3d126d43e9be32731e708ff76c [term 1 FOLLOWER]: Committing config change with OpId 1.94: config changed from index -1 to 94, NON_VOTER 953c332688d14831b3bc7cfd895e06e9 (127.29.124.132) added. New config: { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: NON_VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: true } } }
I20240501 14:04:37.365161 1050 catalog_manager.cc:5484] T 8979fd4d21814622ba687edb88fa7851 P 05f4ef3d126d43e9be32731e708ff76c reported cstate change: config changed from index -1 to 94, NON_VOTER 953c332688d14831b3bc7cfd895e06e9 (127.29.124.132) added. New cstate: current_term: 1 leader_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" committed_config { opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: NON_VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: true } } }
I20240501 14:04:37.460676 3187 tablet_copy_client.cc:308] T c586d994d2414073b25a8b3fab6a7046 P 953c332688d14831b3bc7cfd895e06e9: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.129:40925
I20240501 14:04:37.491397 1452 consensus_queue.cc:237] T 14e026c65736401fb3aec7e6462ae001 P eb004c2ac9d244f89b61448584d4115b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 76, Committed index: 76, Last appended: 1.76, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "32064197afde4b79bf388b080c365d88" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 37211 } attrs { promote: true } }
I20240501 14:04:37.493768 1191 raft_consensus.cc:1261] T 14e026c65736401fb3aec7e6462ae001 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 FOLLOWER]: Refusing update from remote peer eb004c2ac9d244f89b61448584d4115b: Log matching property violated. Preceding OpId in replica: term: 1 index: 76. Preceding OpId from leader: term: 1 index: 77. (index mismatch)
I20240501 14:04:37.500322 2970 consensus_queue.cc:1035] T 14e026c65736401fb3aec7e6462ae001 P eb004c2ac9d244f89b61448584d4115b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 77, Last known committed idx: 76, Time since last communication: 0.000s
I20240501 14:04:37.502925 3013 raft_consensus.cc:2932] T 14e026c65736401fb3aec7e6462ae001 P eb004c2ac9d244f89b61448584d4115b [term 1 LEADER]: Committing config change with OpId 1.77: config changed from index -1 to 77, NON_VOTER 32064197afde4b79bf388b080c365d88 (127.29.124.134) added. New config: { opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "32064197afde4b79bf388b080c365d88" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 37211 } attrs { promote: true } } }
I20240501 14:04:37.503561 1191 raft_consensus.cc:2932] T 14e026c65736401fb3aec7e6462ae001 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 FOLLOWER]: Committing config change with OpId 1.77: config changed from index -1 to 77, NON_VOTER 32064197afde4b79bf388b080c365d88 (127.29.124.134) added. New config: { opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "32064197afde4b79bf388b080c365d88" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 37211 } attrs { promote: true } } }
W20240501 14:04:37.509171 1386 consensus_peers.cc:485] T 14e026c65736401fb3aec7e6462ae001 P eb004c2ac9d244f89b61448584d4115b -> Peer 32064197afde4b79bf388b080c365d88 (127.29.124.134:37211): Couldn't send request to peer 32064197afde4b79bf388b080c365d88. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 14e026c65736401fb3aec7e6462ae001. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:04:37.509701 1045 catalog_manager.cc:5484] T 14e026c65736401fb3aec7e6462ae001 P eb004c2ac9d244f89b61448584d4115b reported cstate change: config changed from index -1 to 77, NON_VOTER 32064197afde4b79bf388b080c365d88 (127.29.124.134) added. New cstate: current_term: 1 leader_uuid: "eb004c2ac9d244f89b61448584d4115b" committed_config { opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32064197afde4b79bf388b080c365d88" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 37211 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:04:37.523736 3204 ts_tablet_manager.cc:902] T 1074c96bc2564edca376f95bd195a62c P 32064197afde4b79bf388b080c365d88: Initiating tablet copy from peer 5d45680f41c84ac39bd97c6f00ac2b06 (127.29.124.129:40925)
I20240501 14:04:37.547189 1213 tablet_copy_service.cc:140] P 5d45680f41c84ac39bd97c6f00ac2b06: Received BeginTabletCopySession request for tablet c586d994d2414073b25a8b3fab6a7046 from peer 953c332688d14831b3bc7cfd895e06e9 ({username='slave'} at 127.29.124.132:34073)
I20240501 14:04:37.547482 1213 tablet_copy_service.cc:161] P 5d45680f41c84ac39bd97c6f00ac2b06: Beginning new tablet copy session on tablet c586d994d2414073b25a8b3fab6a7046 from peer 953c332688d14831b3bc7cfd895e06e9 at {username='slave'} at 127.29.124.132:34073: session id = 953c332688d14831b3bc7cfd895e06e9-c586d994d2414073b25a8b3fab6a7046
I20240501 14:04:37.551993 1213 tablet_copy_source_session.cc:205] T c586d994d2414073b25a8b3fab6a7046 P 5d45680f41c84ac39bd97c6f00ac2b06: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:04:37.553349 3204 tablet_copy_client.cc:308] T 1074c96bc2564edca376f95bd195a62c P 32064197afde4b79bf388b080c365d88: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.129:40925
I20240501 14:04:37.558332 3187 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c586d994d2414073b25a8b3fab6a7046. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:37.587117 3187 tablet_copy_client.cc:783] T c586d994d2414073b25a8b3fab6a7046 P 953c332688d14831b3bc7cfd895e06e9: tablet copy: Starting download of 0 data blocks...
I20240501 14:04:37.587787 3187 tablet_copy_client.cc:647] T c586d994d2414073b25a8b3fab6a7046 P 953c332688d14831b3bc7cfd895e06e9: tablet copy: Starting download of 1 WAL segments...
I20240501 14:04:37.600374 3187 tablet_copy_client.cc:523] T c586d994d2414073b25a8b3fab6a7046 P 953c332688d14831b3bc7cfd895e06e9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:04:37.606676 3187 tablet_bootstrap.cc:492] T c586d994d2414073b25a8b3fab6a7046 P 953c332688d14831b3bc7cfd895e06e9: Bootstrap starting.
I20240501 14:04:37.626482 1192 consensus_queue.cc:237] T 0f837cac32ad4aaa970124639138c2db P 5d45680f41c84ac39bd97c6f00ac2b06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 88, Committed index: 88, Last appended: 1.88, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: true } }
I20240501 14:04:37.629695 1450 raft_consensus.cc:1261] T 0f837cac32ad4aaa970124639138c2db P eb004c2ac9d244f89b61448584d4115b [term 1 FOLLOWER]: Refusing update from remote peer 5d45680f41c84ac39bd97c6f00ac2b06: Log matching property violated. Preceding OpId in replica: term: 1 index: 88. Preceding OpId from leader: term: 1 index: 89. (index mismatch)
I20240501 14:04:37.635574 3170 consensus_queue.cc:1035] T 0f837cac32ad4aaa970124639138c2db P 5d45680f41c84ac39bd97c6f00ac2b06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 89, Last known committed idx: 88, Time since last communication: 0.000s
I20240501 14:04:37.649036 3214 ts_tablet_manager.cc:902] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 752bc4569da647679d5df51641961840: Initiating tablet copy from peer 05f4ef3d126d43e9be32731e708ff76c (127.29.124.130:41511)
I20240501 14:04:37.650964 3217 ts_tablet_manager.cc:902] T 0f11914838ca4139a1c1813fc5334782 P 4e6923f9753a4450ac4343c34c427d1e: Initiating tablet copy from peer 5d45680f41c84ac39bd97c6f00ac2b06 (127.29.124.129:40925)
I20240501 14:04:37.657123 3170 raft_consensus.cc:2932] T 0f837cac32ad4aaa970124639138c2db P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 LEADER]: Committing config change with OpId 1.89: config changed from index -1 to 89, NON_VOTER 752bc4569da647679d5df51641961840 (127.29.124.135) added. New config: { opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: true } } }
I20240501 14:04:37.661659 1213 tablet_copy_service.cc:140] P 5d45680f41c84ac39bd97c6f00ac2b06: Received BeginTabletCopySession request for tablet 1074c96bc2564edca376f95bd195a62c from peer 32064197afde4b79bf388b080c365d88 ({username='slave'} at 127.29.124.134:41391)
I20240501 14:04:37.661834 1213 tablet_copy_service.cc:161] P 5d45680f41c84ac39bd97c6f00ac2b06: Beginning new tablet copy session on tablet 1074c96bc2564edca376f95bd195a62c from peer 32064197afde4b79bf388b080c365d88 at {username='slave'} at 127.29.124.134:41391: session id = 32064197afde4b79bf388b080c365d88-1074c96bc2564edca376f95bd195a62c
I20240501 14:04:37.664777 1213 tablet_copy_source_session.cc:205] T 1074c96bc2564edca376f95bd195a62c P 5d45680f41c84ac39bd97c6f00ac2b06: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:04:37.667289 1450 raft_consensus.cc:2932] T 0f837cac32ad4aaa970124639138c2db P eb004c2ac9d244f89b61448584d4115b [term 1 FOLLOWER]: Committing config change with OpId 1.89: config changed from index -1 to 89, NON_VOTER 752bc4569da647679d5df51641961840 (127.29.124.135) added. New config: { opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: true } } }
I20240501 14:04:37.673647 1050 catalog_manager.cc:5484] T 0f837cac32ad4aaa970124639138c2db P eb004c2ac9d244f89b61448584d4115b reported cstate change: config changed from index -1 to 89, NON_VOTER 752bc4569da647679d5df51641961840 (127.29.124.135) added. New cstate: current_term: 1 leader_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" committed_config { opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: true } } }
I20240501 14:04:37.678324 3204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1074c96bc2564edca376f95bd195a62c. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 14:04:37.680822 1128 consensus_peers.cc:485] T 0f837cac32ad4aaa970124639138c2db P 5d45680f41c84ac39bd97c6f00ac2b06 -> Peer 752bc4569da647679d5df51641961840 (127.29.124.135:42923): Couldn't send request to peer 752bc4569da647679d5df51641961840. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f837cac32ad4aaa970124639138c2db. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:04:37.709359 3217 tablet_copy_client.cc:308] T 0f11914838ca4139a1c1813fc5334782 P 4e6923f9753a4450ac4343c34c427d1e: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.129:40925
I20240501 14:04:37.733207 3204 tablet_copy_client.cc:783] T 1074c96bc2564edca376f95bd195a62c P 32064197afde4b79bf388b080c365d88: tablet copy: Starting download of 0 data blocks...
I20240501 14:04:37.733772 3204 tablet_copy_client.cc:647] T 1074c96bc2564edca376f95bd195a62c P 32064197afde4b79bf388b080c365d88: tablet copy: Starting download of 1 WAL segments...
I20240501 14:04:37.743628 3204 tablet_copy_client.cc:523] T 1074c96bc2564edca376f95bd195a62c P 32064197afde4b79bf388b080c365d88: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:04:37.746964 3187 log.cc:824] T c586d994d2414073b25a8b3fab6a7046 P 953c332688d14831b3bc7cfd895e06e9: Log is configured to *not* fsync() on all Append() calls
I20240501 14:04:37.748668 1213 tablet_copy_service.cc:140] P 5d45680f41c84ac39bd97c6f00ac2b06: Received BeginTabletCopySession request for tablet 0f11914838ca4139a1c1813fc5334782 from peer 4e6923f9753a4450ac4343c34c427d1e ({username='slave'} at 127.29.124.133:36125)
I20240501 14:04:37.748828 1213 tablet_copy_service.cc:161] P 5d45680f41c84ac39bd97c6f00ac2b06: Beginning new tablet copy session on tablet 0f11914838ca4139a1c1813fc5334782 from peer 4e6923f9753a4450ac4343c34c427d1e at {username='slave'} at 127.29.124.133:36125: session id = 4e6923f9753a4450ac4343c34c427d1e-0f11914838ca4139a1c1813fc5334782
I20240501 14:04:37.751156 3204 tablet_bootstrap.cc:492] T 1074c96bc2564edca376f95bd195a62c P 32064197afde4b79bf388b080c365d88: Bootstrap starting.
I20240501 14:04:37.751924 1213 tablet_copy_source_session.cc:205] T 0f11914838ca4139a1c1813fc5334782 P 5d45680f41c84ac39bd97c6f00ac2b06: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:04:37.754611 3217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f11914838ca4139a1c1813fc5334782. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:37.764525 3217 tablet_copy_client.cc:783] T 0f11914838ca4139a1c1813fc5334782 P 4e6923f9753a4450ac4343c34c427d1e: tablet copy: Starting download of 0 data blocks...
I20240501 14:04:37.764937 3217 tablet_copy_client.cc:647] T 0f11914838ca4139a1c1813fc5334782 P 4e6923f9753a4450ac4343c34c427d1e: tablet copy: Starting download of 1 WAL segments...
I20240501 14:04:37.771685 3217 tablet_copy_client.cc:523] T 0f11914838ca4139a1c1813fc5334782 P 4e6923f9753a4450ac4343c34c427d1e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:04:37.791193 3217 tablet_bootstrap.cc:492] T 0f11914838ca4139a1c1813fc5334782 P 4e6923f9753a4450ac4343c34c427d1e: Bootstrap starting.
I20240501 14:04:37.821002 3214 tablet_copy_client.cc:308] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 752bc4569da647679d5df51641961840: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.130:41511
I20240501 14:04:37.831149 1453 consensus_queue.cc:237] T 45d428dc499a4270b56a579f5b3cad83 P eb004c2ac9d244f89b61448584d4115b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 71, Committed index: 71, Last appended: 1.71, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: true } }
I20240501 14:04:37.834637 1323 raft_consensus.cc:1261] T 45d428dc499a4270b56a579f5b3cad83 P 05f4ef3d126d43e9be32731e708ff76c [term 1 FOLLOWER]: Refusing update from remote peer eb004c2ac9d244f89b61448584d4115b: Log matching property violated. Preceding OpId in replica: term: 1 index: 71. Preceding OpId from leader: term: 1 index: 72. (index mismatch)
I20240501 14:04:37.836535 3013 consensus_queue.cc:1035] T 45d428dc499a4270b56a579f5b3cad83 P eb004c2ac9d244f89b61448584d4115b [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 72, Last known committed idx: 71, Time since last communication: 0.000s
I20240501 14:04:37.838964 3013 raft_consensus.cc:2932] T 45d428dc499a4270b56a579f5b3cad83 P eb004c2ac9d244f89b61448584d4115b [term 1 LEADER]: Committing config change with OpId 1.72: config changed from index -1 to 72, NON_VOTER 4e6923f9753a4450ac4343c34c427d1e (127.29.124.133) added. New config: { opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: true } } }
I20240501 14:04:37.839344 1323 raft_consensus.cc:2932] T 45d428dc499a4270b56a579f5b3cad83 P 05f4ef3d126d43e9be32731e708ff76c [term 1 FOLLOWER]: Committing config change with OpId 1.72: config changed from index -1 to 72, NON_VOTER 4e6923f9753a4450ac4343c34c427d1e (127.29.124.133) added. New config: { opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: true } } }
I20240501 14:04:37.844725 1045 catalog_manager.cc:5484] T 45d428dc499a4270b56a579f5b3cad83 P 05f4ef3d126d43e9be32731e708ff76c reported cstate change: config changed from index -1 to 72, NON_VOTER 4e6923f9753a4450ac4343c34c427d1e (127.29.124.133) added. New cstate: current_term: 1 leader_uuid: "eb004c2ac9d244f89b61448584d4115b" committed_config { opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: true } } }
I20240501 14:04:37.860798 3204 log.cc:824] T 1074c96bc2564edca376f95bd195a62c P 32064197afde4b79bf388b080c365d88: Log is configured to *not* fsync() on all Append() calls
I20240501 14:04:37.884367 3238 ts_tablet_manager.cc:902] T 8979fd4d21814622ba687edb88fa7851 P 953c332688d14831b3bc7cfd895e06e9: Initiating tablet copy from peer 5d45680f41c84ac39bd97c6f00ac2b06 (127.29.124.129:40925)
I20240501 14:04:37.902786 1343 tablet_copy_service.cc:140] P 05f4ef3d126d43e9be32731e708ff76c: Received BeginTabletCopySession request for tablet 3a4ca3f4edcd4bf29d0bab40a8856039 from peer 752bc4569da647679d5df51641961840 ({username='slave'} at 127.29.124.135:55917)
I20240501 14:04:37.903056 1343 tablet_copy_service.cc:161] P 05f4ef3d126d43e9be32731e708ff76c: Beginning new tablet copy session on tablet 3a4ca3f4edcd4bf29d0bab40a8856039 from peer 752bc4569da647679d5df51641961840 at {username='slave'} at 127.29.124.135:55917: session id = 752bc4569da647679d5df51641961840-3a4ca3f4edcd4bf29d0bab40a8856039
I20240501 14:04:37.907215 1343 tablet_copy_source_session.cc:205] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 05f4ef3d126d43e9be32731e708ff76c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:04:37.910210 3214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a4ca3f4edcd4bf29d0bab40a8856039. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:37.921972 3214 tablet_copy_client.cc:783] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 752bc4569da647679d5df51641961840: tablet copy: Starting download of 0 data blocks...
I20240501 14:04:37.922355 3214 tablet_copy_client.cc:647] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 752bc4569da647679d5df51641961840: tablet copy: Starting download of 1 WAL segments...
I20240501 14:04:37.926442 3214 tablet_copy_client.cc:523] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 752bc4569da647679d5df51641961840: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240501 14:04:37.926506 1386 consensus_peers.cc:485] T 45d428dc499a4270b56a579f5b3cad83 P eb004c2ac9d244f89b61448584d4115b -> Peer 4e6923f9753a4450ac4343c34c427d1e (127.29.124.133:34675): Couldn't send request to peer 4e6923f9753a4450ac4343c34c427d1e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 45d428dc499a4270b56a579f5b3cad83. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:04:37.926559 3238 tablet_copy_client.cc:308] T 8979fd4d21814622ba687edb88fa7851 P 953c332688d14831b3bc7cfd895e06e9: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.129:40925
I20240501 14:04:37.927928 1213 tablet_copy_service.cc:140] P 5d45680f41c84ac39bd97c6f00ac2b06: Received BeginTabletCopySession request for tablet 8979fd4d21814622ba687edb88fa7851 from peer 953c332688d14831b3bc7cfd895e06e9 ({username='slave'} at 127.29.124.132:34073)
I20240501 14:04:37.928109 1213 tablet_copy_service.cc:161] P 5d45680f41c84ac39bd97c6f00ac2b06: Beginning new tablet copy session on tablet 8979fd4d21814622ba687edb88fa7851 from peer 953c332688d14831b3bc7cfd895e06e9 at {username='slave'} at 127.29.124.132:34073: session id = 953c332688d14831b3bc7cfd895e06e9-8979fd4d21814622ba687edb88fa7851
I20240501 14:04:37.931123 1213 tablet_copy_source_session.cc:205] T 8979fd4d21814622ba687edb88fa7851 P 5d45680f41c84ac39bd97c6f00ac2b06: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:04:37.935364 3238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8979fd4d21814622ba687edb88fa7851. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:37.937961 3214 tablet_bootstrap.cc:492] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 752bc4569da647679d5df51641961840: Bootstrap starting.
I20240501 14:04:37.941718 3238 tablet_copy_client.cc:783] T 8979fd4d21814622ba687edb88fa7851 P 953c332688d14831b3bc7cfd895e06e9: tablet copy: Starting download of 0 data blocks...
I20240501 14:04:37.942018 3238 tablet_copy_client.cc:647] T 8979fd4d21814622ba687edb88fa7851 P 953c332688d14831b3bc7cfd895e06e9: tablet copy: Starting download of 1 WAL segments...
I20240501 14:04:37.950232 3238 tablet_copy_client.cc:523] T 8979fd4d21814622ba687edb88fa7851 P 953c332688d14831b3bc7cfd895e06e9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:04:37.956737 3238 tablet_bootstrap.cc:492] T 8979fd4d21814622ba687edb88fa7851 P 953c332688d14831b3bc7cfd895e06e9: Bootstrap starting.
I20240501 14:04:37.958949 1451 consensus_queue.cc:237] T 6a5325b7aa824831a5fa056fb4439110 P eb004c2ac9d244f89b61448584d4115b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 78, Committed index: 78, Last appended: 1.78, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: NON_VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: true } }
I20240501 14:04:37.984263 1323 raft_consensus.cc:1261] T 6a5325b7aa824831a5fa056fb4439110 P 05f4ef3d126d43e9be32731e708ff76c [term 1 FOLLOWER]: Refusing update from remote peer eb004c2ac9d244f89b61448584d4115b: Log matching property violated. Preceding OpId in replica: term: 1 index: 78. Preceding OpId from leader: term: 1 index: 79. (index mismatch)
I20240501 14:04:37.984874 2970 consensus_queue.cc:1035] T 6a5325b7aa824831a5fa056fb4439110 P eb004c2ac9d244f89b61448584d4115b [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 79, Last known committed idx: 78, Time since last communication: 0.000s
I20240501 14:04:37.987313 2970 raft_consensus.cc:2932] T 6a5325b7aa824831a5fa056fb4439110 P eb004c2ac9d244f89b61448584d4115b [term 1 LEADER]: Committing config change with OpId 1.79: config changed from index -1 to 79, NON_VOTER 953c332688d14831b3bc7cfd895e06e9 (127.29.124.132) added. New config: { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: NON_VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: true } } }
I20240501 14:04:37.997551 1322 raft_consensus.cc:2932] T 6a5325b7aa824831a5fa056fb4439110 P 05f4ef3d126d43e9be32731e708ff76c [term 1 FOLLOWER]: Committing config change with OpId 1.79: config changed from index -1 to 79, NON_VOTER 953c332688d14831b3bc7cfd895e06e9 (127.29.124.132) added. New config: { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: NON_VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: true } } }
I20240501 14:04:38.007078 3249 ts_tablet_manager.cc:902] T 14e026c65736401fb3aec7e6462ae001 P 32064197afde4b79bf388b080c365d88: Initiating tablet copy from peer eb004c2ac9d244f89b61448584d4115b (127.29.124.131:40225)
I20240501 14:04:38.007313 1045 catalog_manager.cc:5484] T 6a5325b7aa824831a5fa056fb4439110 P eb004c2ac9d244f89b61448584d4115b reported cstate change: config changed from index -1 to 79, NON_VOTER 953c332688d14831b3bc7cfd895e06e9 (127.29.124.132) added. New cstate: current_term: 1 leader_uuid: "eb004c2ac9d244f89b61448584d4115b" committed_config { opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: NON_VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 14:04:38.019495 1388 consensus_peers.cc:485] T 6a5325b7aa824831a5fa056fb4439110 P eb004c2ac9d244f89b61448584d4115b -> Peer 953c332688d14831b3bc7cfd895e06e9 (127.29.124.132:34305): Couldn't send request to peer 953c332688d14831b3bc7cfd895e06e9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6a5325b7aa824831a5fa056fb4439110. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:04:38.088574 3187 tablet_bootstrap.cc:492] T c586d994d2414073b25a8b3fab6a7046 P 953c332688d14831b3bc7cfd895e06e9: Bootstrap replayed 1/1 log segments. Stats: ops{read=57 overwritten=0 applied=57 ignored=0} inserts{seen=688 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:04:38.100050 3187 tablet_bootstrap.cc:492] T c586d994d2414073b25a8b3fab6a7046 P 953c332688d14831b3bc7cfd895e06e9: Bootstrap complete.
I20240501 14:04:38.101063 3187 ts_tablet_manager.cc:1367] T c586d994d2414073b25a8b3fab6a7046 P 953c332688d14831b3bc7cfd895e06e9: Time spent bootstrapping tablet: real 0.495s user 0.107s sys 0.020s
I20240501 14:04:38.112946 3187 raft_consensus.cc:348] T c586d994d2414073b25a8b3fab6a7046 P 953c332688d14831b3bc7cfd895e06e9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 47 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: NON_VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: true } }
I20240501 14:04:38.117477 3187 raft_consensus.cc:729] T c586d994d2414073b25a8b3fab6a7046 P 953c332688d14831b3bc7cfd895e06e9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 953c332688d14831b3bc7cfd895e06e9, State: Initialized, Role: LEARNER
I20240501 14:04:38.118419 3187 consensus_queue.cc:260] T c586d994d2414073b25a8b3fab6a7046 P 953c332688d14831b3bc7cfd895e06e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 57, Last appended: 1.57, Last appended by leader: 57, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 47 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: NON_VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: true } }
I20240501 14:04:38.119596 3260 ts_tablet_manager.cc:902] T 0f837cac32ad4aaa970124639138c2db P 752bc4569da647679d5df51641961840: Initiating tablet copy from peer 5d45680f41c84ac39bd97c6f00ac2b06 (127.29.124.129:40925)
I20240501 14:04:38.129151 3187 ts_tablet_manager.cc:1398] T c586d994d2414073b25a8b3fab6a7046 P 953c332688d14831b3bc7cfd895e06e9: Time spent starting tablet: real 0.027s user 0.007s sys 0.004s
I20240501 14:04:38.133653 1213 tablet_copy_service.cc:342] P 5d45680f41c84ac39bd97c6f00ac2b06: Request end of tablet copy session 953c332688d14831b3bc7cfd895e06e9-c586d994d2414073b25a8b3fab6a7046 received from {username='slave'} at 127.29.124.132:34073
I20240501 14:04:38.133798 1213 tablet_copy_service.cc:433] P 5d45680f41c84ac39bd97c6f00ac2b06: ending tablet copy session 953c332688d14831b3bc7cfd895e06e9-c586d994d2414073b25a8b3fab6a7046 on tablet c586d994d2414073b25a8b3fab6a7046 with peer 953c332688d14831b3bc7cfd895e06e9
I20240501 14:04:38.149369 3249 tablet_copy_client.cc:308] T 14e026c65736401fb3aec7e6462ae001 P 32064197afde4b79bf388b080c365d88: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.131:40225
I20240501 14:04:38.158226 1473 tablet_copy_service.cc:140] P eb004c2ac9d244f89b61448584d4115b: Received BeginTabletCopySession request for tablet 14e026c65736401fb3aec7e6462ae001 from peer 32064197afde4b79bf388b080c365d88 ({username='slave'} at 127.29.124.134:43151)
I20240501 14:04:38.158529 1473 tablet_copy_service.cc:161] P eb004c2ac9d244f89b61448584d4115b: Beginning new tablet copy session on tablet 14e026c65736401fb3aec7e6462ae001 from peer 32064197afde4b79bf388b080c365d88 at {username='slave'} at 127.29.124.134:43151: session id = 32064197afde4b79bf388b080c365d88-14e026c65736401fb3aec7e6462ae001
I20240501 14:04:38.162178 1473 tablet_copy_source_session.cc:205] T 14e026c65736401fb3aec7e6462ae001 P eb004c2ac9d244f89b61448584d4115b: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:04:38.166249 3249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14e026c65736401fb3aec7e6462ae001. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:38.174341 3249 tablet_copy_client.cc:783] T 14e026c65736401fb3aec7e6462ae001 P 32064197afde4b79bf388b080c365d88: tablet copy: Starting download of 0 data blocks...
I20240501 14:04:38.174710 3249 tablet_copy_client.cc:647] T 14e026c65736401fb3aec7e6462ae001 P 32064197afde4b79bf388b080c365d88: tablet copy: Starting download of 1 WAL segments...
I20240501 14:04:38.176929 3249 tablet_copy_client.cc:523] T 14e026c65736401fb3aec7e6462ae001 P 32064197afde4b79bf388b080c365d88: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:04:38.176951 1193 consensus_queue.cc:237] T 70afcb72943b49a7bea962791b38a91d P 5d45680f41c84ac39bd97c6f00ac2b06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 82, Committed index: 82, Last appended: 1.83, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } } peers { permanent_uuid: "32064197afde4b79bf388b080c365d88" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 37211 } attrs { promote: true } }
W20240501 14:04:38.178817 1126 consensus_peers.cc:485] T 70afcb72943b49a7bea962791b38a91d P 5d45680f41c84ac39bd97c6f00ac2b06 -> Peer 32064197afde4b79bf388b080c365d88 (127.29.124.134:37211): Couldn't send request to peer 32064197afde4b79bf388b080c365d88. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 70afcb72943b49a7bea962791b38a91d. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:04:38.180907 3249 tablet_bootstrap.cc:492] T 14e026c65736401fb3aec7e6462ae001 P 32064197afde4b79bf388b080c365d88: Bootstrap starting.
I20240501 14:04:38.190588 1453 raft_consensus.cc:1261] T 70afcb72943b49a7bea962791b38a91d P eb004c2ac9d244f89b61448584d4115b [term 1 FOLLOWER]: Refusing update from remote peer 5d45680f41c84ac39bd97c6f00ac2b06: Log matching property violated. Preceding OpId in replica: term: 1 index: 83. Preceding OpId from leader: term: 1 index: 84. (index mismatch)
I20240501 14:04:38.191872 3153 consensus_queue.cc:1035] T 70afcb72943b49a7bea962791b38a91d P 5d45680f41c84ac39bd97c6f00ac2b06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 84, Last known committed idx: 82, Time since last communication: 0.000s
I20240501 14:04:38.205657 3170 raft_consensus.cc:2932] T 70afcb72943b49a7bea962791b38a91d P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 LEADER]: Committing config change with OpId 1.84: config changed from index -1 to 84, NON_VOTER 32064197afde4b79bf388b080c365d88 (127.29.124.134) added. New config: { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } } peers { permanent_uuid: "32064197afde4b79bf388b080c365d88" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 37211 } attrs { promote: true } } }
I20240501 14:04:38.210419 1045 catalog_manager.cc:5484] T 70afcb72943b49a7bea962791b38a91d P 5d45680f41c84ac39bd97c6f00ac2b06 reported cstate change: config changed from index -1 to 84, NON_VOTER 32064197afde4b79bf388b080c365d88 (127.29.124.134) added. New cstate: current_term: 1 leader_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" committed_config { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32064197afde4b79bf388b080c365d88" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 37211 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:04:38.238044 3269 ts_tablet_manager.cc:902] T 45d428dc499a4270b56a579f5b3cad83 P 4e6923f9753a4450ac4343c34c427d1e: Initiating tablet copy from peer eb004c2ac9d244f89b61448584d4115b (127.29.124.131:40225)
I20240501 14:04:38.241747 1452 raft_consensus.cc:2932] T 70afcb72943b49a7bea962791b38a91d P eb004c2ac9d244f89b61448584d4115b [term 1 FOLLOWER]: Committing config change with OpId 1.84: config changed from index -1 to 84, NON_VOTER 32064197afde4b79bf388b080c365d88 (127.29.124.134) added. New config: { opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } } peers { permanent_uuid: "32064197afde4b79bf388b080c365d88" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 37211 } attrs { promote: true } } }
I20240501 14:04:38.296198 3260 tablet_copy_client.cc:308] T 0f837cac32ad4aaa970124639138c2db P 752bc4569da647679d5df51641961840: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.129:40925
I20240501 14:04:38.313587 1213 tablet_copy_service.cc:140] P 5d45680f41c84ac39bd97c6f00ac2b06: Received BeginTabletCopySession request for tablet 0f837cac32ad4aaa970124639138c2db from peer 752bc4569da647679d5df51641961840 ({username='slave'} at 127.29.124.135:47221)
I20240501 14:04:38.313763 1213 tablet_copy_service.cc:161] P 5d45680f41c84ac39bd97c6f00ac2b06: Beginning new tablet copy session on tablet 0f837cac32ad4aaa970124639138c2db from peer 752bc4569da647679d5df51641961840 at {username='slave'} at 127.29.124.135:47221: session id = 752bc4569da647679d5df51641961840-0f837cac32ad4aaa970124639138c2db
I20240501 14:04:38.316816 1213 tablet_copy_source_session.cc:205] T 0f837cac32ad4aaa970124639138c2db P 5d45680f41c84ac39bd97c6f00ac2b06: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:04:38.321589 3238 tablet_bootstrap.cc:492] T 8979fd4d21814622ba687edb88fa7851 P 953c332688d14831b3bc7cfd895e06e9: Bootstrap replayed 1/1 log segments. Stats: ops{read=107 overwritten=0 applied=107 ignored=0} inserts{seen=738 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:04:38.322072 3238 tablet_bootstrap.cc:492] T 8979fd4d21814622ba687edb88fa7851 P 953c332688d14831b3bc7cfd895e06e9: Bootstrap complete.
I20240501 14:04:38.322387 3238 ts_tablet_manager.cc:1367] T 8979fd4d21814622ba687edb88fa7851 P 953c332688d14831b3bc7cfd895e06e9: Time spent bootstrapping tablet: real 0.366s user 0.164s sys 0.045s
I20240501 14:04:38.328104 3260 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f837cac32ad4aaa970124639138c2db. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:38.328199 3238 raft_consensus.cc:348] T 8979fd4d21814622ba687edb88fa7851 P 953c332688d14831b3bc7cfd895e06e9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: NON_VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: true } }
I20240501 14:04:38.328524 3238 raft_consensus.cc:729] T 8979fd4d21814622ba687edb88fa7851 P 953c332688d14831b3bc7cfd895e06e9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 953c332688d14831b3bc7cfd895e06e9, State: Initialized, Role: LEARNER
I20240501 14:04:38.328758 3238 consensus_queue.cc:260] T 8979fd4d21814622ba687edb88fa7851 P 953c332688d14831b3bc7cfd895e06e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 107, Last appended: 1.107, Last appended by leader: 107, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 94 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: NON_VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: true } }
I20240501 14:04:38.330338 3238 ts_tablet_manager.cc:1398] T 8979fd4d21814622ba687edb88fa7851 P 953c332688d14831b3bc7cfd895e06e9: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20240501 14:04:38.332793 1213 tablet_copy_service.cc:342] P 5d45680f41c84ac39bd97c6f00ac2b06: Request end of tablet copy session 953c332688d14831b3bc7cfd895e06e9-8979fd4d21814622ba687edb88fa7851 received from {username='slave'} at 127.29.124.132:34073
I20240501 14:04:38.332935 1213 tablet_copy_service.cc:433] P 5d45680f41c84ac39bd97c6f00ac2b06: ending tablet copy session 953c332688d14831b3bc7cfd895e06e9-8979fd4d21814622ba687edb88fa7851 on tablet 8979fd4d21814622ba687edb88fa7851 with peer 953c332688d14831b3bc7cfd895e06e9
I20240501 14:04:38.334388 2489 raft_consensus.cc:1203] T 8979fd4d21814622ba687edb88fa7851 P 953c332688d14831b3bc7cfd895e06e9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.93->[1.94-1.108] Dedup: 1.107->[1.108-1.108]
I20240501 14:04:38.399206 3217 tablet_bootstrap.cc:492] T 0f11914838ca4139a1c1813fc5334782 P 4e6923f9753a4450ac4343c34c427d1e: Bootstrap replayed 1/1 log segments. Stats: ops{read=57 overwritten=0 applied=57 ignored=0} inserts{seen=688 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:04:38.407066 3217 tablet_bootstrap.cc:492] T 0f11914838ca4139a1c1813fc5334782 P 4e6923f9753a4450ac4343c34c427d1e: Bootstrap complete.
I20240501 14:04:38.408208 3217 ts_tablet_manager.cc:1367] T 0f11914838ca4139a1c1813fc5334782 P 4e6923f9753a4450ac4343c34c427d1e: Time spent bootstrapping tablet: real 0.617s user 0.097s sys 0.028s
I20240501 14:04:38.410001 3217 raft_consensus.cc:348] T 0f11914838ca4139a1c1813fc5334782 P 4e6923f9753a4450ac4343c34c427d1e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: true } }
I20240501 14:04:38.412431 3065 raft_consensus.cc:1050] T 8979fd4d21814622ba687edb88fa7851 P 5d45680f41c84ac39bd97c6f00ac2b06: attempting to promote NON_VOTER 953c332688d14831b3bc7cfd895e06e9 to VOTER
I20240501 14:04:38.413374 3065 consensus_queue.cc:237] T 8979fd4d21814622ba687edb88fa7851 P 5d45680f41c84ac39bd97c6f00ac2b06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 109, Committed index: 109, Last appended: 1.109, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } }
I20240501 14:04:38.413996 3217 raft_consensus.cc:729] T 0f11914838ca4139a1c1813fc5334782 P 4e6923f9753a4450ac4343c34c427d1e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4e6923f9753a4450ac4343c34c427d1e, State: Initialized, Role: LEARNER
I20240501 14:04:38.414242 3217 consensus_queue.cc:260] T 0f11914838ca4139a1c1813fc5334782 P 4e6923f9753a4450ac4343c34c427d1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 57, Last appended: 1.57, Last appended by leader: 57, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 50 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: true } }
I20240501 14:04:38.420547 3269 tablet_copy_client.cc:308] T 45d428dc499a4270b56a579f5b3cad83 P 4e6923f9753a4450ac4343c34c427d1e: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.131:40225
I20240501 14:04:38.420594 1322 raft_consensus.cc:1261] T 8979fd4d21814622ba687edb88fa7851 P 05f4ef3d126d43e9be32731e708ff76c [term 1 FOLLOWER]: Refusing update from remote peer 5d45680f41c84ac39bd97c6f00ac2b06: Log matching property violated. Preceding OpId in replica: term: 1 index: 109. Preceding OpId from leader: term: 1 index: 110. (index mismatch)
I20240501 14:04:38.421110 2488 raft_consensus.cc:1203] T c586d994d2414073b25a8b3fab6a7046 P 953c332688d14831b3bc7cfd895e06e9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.46->[1.47-1.63] Dedup: 1.57->[1.58-1.63]
I20240501 14:04:38.421144 2392 consensus_queue.cc:1035] T 8979fd4d21814622ba687edb88fa7851 P 5d45680f41c84ac39bd97c6f00ac2b06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 110, Last known committed idx: 109, Time since last communication: 0.000s
I20240501 14:04:38.428557 2489 raft_consensus.cc:1261] T 8979fd4d21814622ba687edb88fa7851 P 953c332688d14831b3bc7cfd895e06e9 [term 1 LEARNER]: Refusing update from remote peer 5d45680f41c84ac39bd97c6f00ac2b06: Log matching property violated. Preceding OpId in replica: term: 1 index: 109. Preceding OpId from leader: term: 1 index: 110. (index mismatch)
I20240501 14:04:38.430568 3170 consensus_queue.cc:1035] T 8979fd4d21814622ba687edb88fa7851 P 5d45680f41c84ac39bd97c6f00ac2b06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 110, Last known committed idx: 109, Time since last communication: 0.000s
I20240501 14:04:38.433107 2392 raft_consensus.cc:2932] T 8979fd4d21814622ba687edb88fa7851 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 LEADER]: Committing config change with OpId 1.110: config changed from index 94 to 110, 953c332688d14831b3bc7cfd895e06e9 (127.29.124.132) changed from NON_VOTER to VOTER. New config: { opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } } }
I20240501 14:04:38.436722 3260 tablet_copy_client.cc:783] T 0f837cac32ad4aaa970124639138c2db P 752bc4569da647679d5df51641961840: tablet copy: Starting download of 0 data blocks...
I20240501 14:04:38.436968 3260 tablet_copy_client.cc:647] T 0f837cac32ad4aaa970124639138c2db P 752bc4569da647679d5df51641961840: tablet copy: Starting download of 1 WAL segments...
I20240501 14:04:38.439090 3260 tablet_copy_client.cc:523] T 0f837cac32ad4aaa970124639138c2db P 752bc4569da647679d5df51641961840: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:04:38.439252 1323 consensus_queue.cc:237] T db2b3f73d20c4790808bbe0f1d5e489c P 05f4ef3d126d43e9be32731e708ff76c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 85, Committed index: 85, Last appended: 1.86, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: true } }
I20240501 14:04:38.441967 2489 raft_consensus.cc:2932] T 8979fd4d21814622ba687edb88fa7851 P 953c332688d14831b3bc7cfd895e06e9 [term 1 FOLLOWER]: Committing config change with OpId 1.110: config changed from index 94 to 110, 953c332688d14831b3bc7cfd895e06e9 (127.29.124.132) changed from NON_VOTER to VOTER. New config: { opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } } }
I20240501 14:04:38.443732 3217 ts_tablet_manager.cc:1398] T 0f11914838ca4139a1c1813fc5334782 P 4e6923f9753a4450ac4343c34c427d1e: Time spent starting tablet: real 0.035s user 0.003s sys 0.000s
W20240501 14:04:38.445842 1258 consensus_peers.cc:485] T db2b3f73d20c4790808bbe0f1d5e489c P 05f4ef3d126d43e9be32731e708ff76c -> Peer 752bc4569da647679d5df51641961840 (127.29.124.135:42923): Couldn't send request to peer 752bc4569da647679d5df51641961840. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: db2b3f73d20c4790808bbe0f1d5e489c. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:04:38.446902 3260 tablet_bootstrap.cc:492] T 0f837cac32ad4aaa970124639138c2db P 752bc4569da647679d5df51641961840: Bootstrap starting.
I20240501 14:04:38.446651 1045 catalog_manager.cc:5484] T 8979fd4d21814622ba687edb88fa7851 P 5d45680f41c84ac39bd97c6f00ac2b06 reported cstate change: config changed from index 94 to 110, 953c332688d14831b3bc7cfd895e06e9 (127.29.124.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" committed_config { opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:04:38.447575 1213 tablet_copy_service.cc:342] P 5d45680f41c84ac39bd97c6f00ac2b06: Request end of tablet copy session 4e6923f9753a4450ac4343c34c427d1e-0f11914838ca4139a1c1813fc5334782 received from {username='slave'} at 127.29.124.133:36125
I20240501 14:04:38.447707 1213 tablet_copy_service.cc:433] P 5d45680f41c84ac39bd97c6f00ac2b06: ending tablet copy session 4e6923f9753a4450ac4343c34c427d1e-0f11914838ca4139a1c1813fc5334782 on tablet 0f11914838ca4139a1c1813fc5334782 with peer 4e6923f9753a4450ac4343c34c427d1e
I20240501 14:04:38.454589 1322 raft_consensus.cc:2932] T 8979fd4d21814622ba687edb88fa7851 P 05f4ef3d126d43e9be32731e708ff76c [term 1 FOLLOWER]: Committing config change with OpId 1.110: config changed from index 94 to 110, 953c332688d14831b3bc7cfd895e06e9 (127.29.124.132) changed from NON_VOTER to VOTER. New config: { opid_index: 110 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } } }
I20240501 14:04:38.455420 1473 tablet_copy_service.cc:140] P eb004c2ac9d244f89b61448584d4115b: Received BeginTabletCopySession request for tablet 45d428dc499a4270b56a579f5b3cad83 from peer 4e6923f9753a4450ac4343c34c427d1e ({username='slave'} at 127.29.124.133:45461)
I20240501 14:04:38.455585 1473 tablet_copy_service.cc:161] P eb004c2ac9d244f89b61448584d4115b: Beginning new tablet copy session on tablet 45d428dc499a4270b56a579f5b3cad83 from peer 4e6923f9753a4450ac4343c34c427d1e at {username='slave'} at 127.29.124.133:45461: session id = 4e6923f9753a4450ac4343c34c427d1e-45d428dc499a4270b56a579f5b3cad83
I20240501 14:04:38.458591 1473 tablet_copy_source_session.cc:205] T 45d428dc499a4270b56a579f5b3cad83 P eb004c2ac9d244f89b61448584d4115b: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:04:38.459832 3269 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 45d428dc499a4270b56a579f5b3cad83. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:38.461360 2622 raft_consensus.cc:1203] T 0f11914838ca4139a1c1813fc5334782 P 4e6923f9753a4450ac4343c34c427d1e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.49->[1.50-1.70] Dedup: 1.57->[1.58-1.70]
I20240501 14:04:38.467279 3269 tablet_copy_client.cc:783] T 45d428dc499a4270b56a579f5b3cad83 P 4e6923f9753a4450ac4343c34c427d1e: tablet copy: Starting download of 0 data blocks...
I20240501 14:04:38.467600 3269 tablet_copy_client.cc:647] T 45d428dc499a4270b56a579f5b3cad83 P 4e6923f9753a4450ac4343c34c427d1e: tablet copy: Starting download of 1 WAL segments...
I20240501 14:04:38.470605 3269 tablet_copy_client.cc:523] T 45d428dc499a4270b56a579f5b3cad83 P 4e6923f9753a4450ac4343c34c427d1e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:04:38.472318 1193 raft_consensus.cc:1261] T db2b3f73d20c4790808bbe0f1d5e489c P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 FOLLOWER]: Refusing update from remote peer 05f4ef3d126d43e9be32731e708ff76c: Log matching property violated. Preceding OpId in replica: term: 1 index: 85. Preceding OpId from leader: term: 1 index: 87. (index mismatch)
I20240501 14:04:38.475879 3269 tablet_bootstrap.cc:492] T 45d428dc499a4270b56a579f5b3cad83 P 4e6923f9753a4450ac4343c34c427d1e: Bootstrap starting.
I20240501 14:04:38.477139 3277 consensus_queue.cc:1035] T db2b3f73d20c4790808bbe0f1d5e489c P 05f4ef3d126d43e9be32731e708ff76c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 87, Last known committed idx: 85, Time since last communication: 0.000s
I20240501 14:04:38.481441 3238 ts_tablet_manager.cc:902] T 6a5325b7aa824831a5fa056fb4439110 P 953c332688d14831b3bc7cfd895e06e9: Initiating tablet copy from peer eb004c2ac9d244f89b61448584d4115b (127.29.124.131:40225)
I20240501 14:04:38.487048 1192 consensus_queue.cc:237] T 8979fd4d21814622ba687edb88fa7851 P 5d45680f41c84ac39bd97c6f00ac2b06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 110, Committed index: 110, Last appended: 1.110, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 111 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } }
I20240501 14:04:38.490379 2489 raft_consensus.cc:1261] T 8979fd4d21814622ba687edb88fa7851 P 953c332688d14831b3bc7cfd895e06e9 [term 1 FOLLOWER]: Refusing update from remote peer 5d45680f41c84ac39bd97c6f00ac2b06: Log matching property violated. Preceding OpId in replica: term: 1 index: 110. Preceding OpId from leader: term: 1 index: 111. (index mismatch)
I20240501 14:04:38.490957 3153 consensus_queue.cc:1035] T 8979fd4d21814622ba687edb88fa7851 P 5d45680f41c84ac39bd97c6f00ac2b06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 111, Last known committed idx: 110, Time since last communication: 0.000s
I20240501 14:04:38.493225 3170 raft_consensus.cc:2932] T 8979fd4d21814622ba687edb88fa7851 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 LEADER]: Committing config change with OpId 1.111: config changed from index 110 to 111, VOTER 05f4ef3d126d43e9be32731e708ff76c (127.29.124.130) evicted. New config: { opid_index: 111 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } } }
I20240501 14:04:38.493814 2489 raft_consensus.cc:2932] T 8979fd4d21814622ba687edb88fa7851 P 953c332688d14831b3bc7cfd895e06e9 [term 1 FOLLOWER]: Committing config change with OpId 1.111: config changed from index 110 to 111, VOTER 05f4ef3d126d43e9be32731e708ff76c (127.29.124.130) evicted. New config: { opid_index: 111 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } } }
I20240501 14:04:38.499493 1045 catalog_manager.cc:5484] T 8979fd4d21814622ba687edb88fa7851 P 5d45680f41c84ac39bd97c6f00ac2b06 reported cstate change: config changed from index 110 to 111, VOTER 05f4ef3d126d43e9be32731e708ff76c (127.29.124.130) evicted. New cstate: current_term: 1 leader_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" committed_config { opid_index: 111 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:04:38.509528 1302 tablet_service.cc:1514] Processing DeleteTablet for tablet 8979fd4d21814622ba687edb88fa7851 with delete_type TABLET_DATA_TOMBSTONED (TS 05f4ef3d126d43e9be32731e708ff76c not found in new config with opid_index 111) from {username='slave'} at 127.0.0.1:35916
I20240501 14:04:38.515553 3204 tablet_bootstrap.cc:492] T 1074c96bc2564edca376f95bd195a62c P 32064197afde4b79bf388b080c365d88: Bootstrap replayed 1/1 log segments. Stats: ops{read=81 overwritten=0 applied=80 ignored=0} inserts{seen=765 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 14:04:38.516287 3204 tablet_bootstrap.cc:492] T 1074c96bc2564edca376f95bd195a62c P 32064197afde4b79bf388b080c365d88: Bootstrap complete.
I20240501 14:04:38.516750 3204 ts_tablet_manager.cc:1367] T 1074c96bc2564edca376f95bd195a62c P 32064197afde4b79bf388b080c365d88: Time spent bootstrapping tablet: real 0.766s user 0.130s sys 0.024s
I20240501 14:04:38.527181 1036 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8979fd4d21814622ba687edb88fa7851 with cas_config_opid_index 110: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:04:38.531248 3295 tablet_replica.cc:330] T 8979fd4d21814622ba687edb88fa7851 P 05f4ef3d126d43e9be32731e708ff76c: stopping tablet replica
I20240501 14:04:38.540498 3046 raft_consensus.cc:2932] T db2b3f73d20c4790808bbe0f1d5e489c P 05f4ef3d126d43e9be32731e708ff76c [term 1 LEADER]: Committing config change with OpId 1.87: config changed from index -1 to 87, NON_VOTER 752bc4569da647679d5df51641961840 (127.29.124.135) added. New config: { opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: true } } }
I20240501 14:04:38.543326 3295 raft_consensus.cc:2229] T 8979fd4d21814622ba687edb88fa7851 P 05f4ef3d126d43e9be32731e708ff76c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:04:38.544114 3295 raft_consensus.cc:2258] T 8979fd4d21814622ba687edb88fa7851 P 05f4ef3d126d43e9be32731e708ff76c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:04:38.549587 1050 catalog_manager.cc:5484] T db2b3f73d20c4790808bbe0f1d5e489c P 05f4ef3d126d43e9be32731e708ff76c reported cstate change: config changed from index -1 to 87, NON_VOTER 752bc4569da647679d5df51641961840 (127.29.124.135) added. New cstate: current_term: 1 leader_uuid: "05f4ef3d126d43e9be32731e708ff76c" committed_config { opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:04:38.550199 3295 ts_tablet_manager.cc:1875] T 8979fd4d21814622ba687edb88fa7851 P 05f4ef3d126d43e9be32731e708ff76c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:04:38.560568 1193 raft_consensus.cc:2932] T db2b3f73d20c4790808bbe0f1d5e489c P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 FOLLOWER]: Committing config change with OpId 1.87: config changed from index -1 to 87, NON_VOTER 752bc4569da647679d5df51641961840 (127.29.124.135) added. New config: { opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: true } } }
I20240501 14:04:38.562716 3170 raft_consensus.cc:1050] T c586d994d2414073b25a8b3fab6a7046 P 5d45680f41c84ac39bd97c6f00ac2b06: attempting to promote NON_VOTER 953c332688d14831b3bc7cfd895e06e9 to VOTER
I20240501 14:04:38.563423 3170 consensus_queue.cc:237] T c586d994d2414073b25a8b3fab6a7046 P 5d45680f41c84ac39bd97c6f00ac2b06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 63, Committed index: 63, Last appended: 1.64, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } }
I20240501 14:04:38.565834 2392 raft_consensus.cc:1050] T 0f11914838ca4139a1c1813fc5334782 P 5d45680f41c84ac39bd97c6f00ac2b06: attempting to promote NON_VOTER 4e6923f9753a4450ac4343c34c427d1e to VOTER
I20240501 14:04:38.566478 2392 consensus_queue.cc:237] T 0f11914838ca4139a1c1813fc5334782 P 5d45680f41c84ac39bd97c6f00ac2b06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 70, Committed index: 70, Last appended: 1.71, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: false } }
I20240501 14:04:38.568419 2622 raft_consensus.cc:1261] T 0f11914838ca4139a1c1813fc5334782 P 4e6923f9753a4450ac4343c34c427d1e [term 1 LEARNER]: Refusing update from remote peer 5d45680f41c84ac39bd97c6f00ac2b06: Log matching property violated. Preceding OpId in replica: term: 1 index: 71. Preceding OpId from leader: term: 1 index: 72. (index mismatch)
I20240501 14:04:38.568573 2489 raft_consensus.cc:1261] T c586d994d2414073b25a8b3fab6a7046 P 953c332688d14831b3bc7cfd895e06e9 [term 1 LEARNER]: Refusing update from remote peer 5d45680f41c84ac39bd97c6f00ac2b06: Log matching property violated. Preceding OpId in replica: term: 1 index: 64. Preceding OpId from leader: term: 1 index: 65. (index mismatch)
I20240501 14:04:38.568888 3170 consensus_queue.cc:1035] T 0f11914838ca4139a1c1813fc5334782 P 5d45680f41c84ac39bd97c6f00ac2b06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 72, Last known committed idx: 70, Time since last communication: 0.000s
I20240501 14:04:38.569682 3170 consensus_queue.cc:1035] T c586d994d2414073b25a8b3fab6a7046 P 5d45680f41c84ac39bd97c6f00ac2b06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 65, Last known committed idx: 63, Time since last communication: 0.000s
I20240501 14:04:38.570864 3295 ts_tablet_manager.cc:1888] T 8979fd4d21814622ba687edb88fa7851 P 05f4ef3d126d43e9be32731e708ff76c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.110
I20240501 14:04:38.560573 3204 raft_consensus.cc:348] T 1074c96bc2564edca376f95bd195a62c P 32064197afde4b79bf388b080c365d88 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "32064197afde4b79bf388b080c365d88" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 37211 } attrs { promote: true } }
I20240501 14:04:38.571326 3153 raft_consensus.cc:2932] T 0f11914838ca4139a1c1813fc5334782 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 LEADER]: Committing config change with OpId 1.72: config changed from index 50 to 72, 4e6923f9753a4450ac4343c34c427d1e (127.29.124.133) changed from NON_VOTER to VOTER. New config: { opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: false } } }
I20240501 14:04:38.574913 3238 tablet_copy_client.cc:308] T 6a5325b7aa824831a5fa056fb4439110 P 953c332688d14831b3bc7cfd895e06e9: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.131:40225
I20240501 14:04:38.575801 3295 log.cc:1196] T 8979fd4d21814622ba687edb88fa7851 P 05f4ef3d126d43e9be32731e708ff76c: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/wal/wals/8979fd4d21814622ba687edb88fa7851
I20240501 14:04:38.576788 1038 catalog_manager.cc:4830] TS 05f4ef3d126d43e9be32731e708ff76c (127.29.124.130:41511): tablet 8979fd4d21814622ba687edb88fa7851 (table rebalance_test_table_0 [id=817d8db2cf8f4cffb29e21f78e86e2d5]) successfully deleted
I20240501 14:04:38.578145 3204 raft_consensus.cc:729] T 1074c96bc2564edca376f95bd195a62c P 32064197afde4b79bf388b080c365d88 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 32064197afde4b79bf388b080c365d88, State: Initialized, Role: LEARNER
I20240501 14:04:38.578992 3065 raft_consensus.cc:2932] T c586d994d2414073b25a8b3fab6a7046 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 LEADER]: Committing config change with OpId 1.65: config changed from index 47 to 65, 953c332688d14831b3bc7cfd895e06e9 (127.29.124.132) changed from NON_VOTER to VOTER. New config: { opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } } }
I20240501 14:04:38.582540 3204 consensus_queue.cc:260] T 1074c96bc2564edca376f95bd195a62c P 32064197afde4b79bf388b080c365d88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 80, Last appended: 1.81, Last appended by leader: 81, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 63 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "32064197afde4b79bf388b080c365d88" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 37211 } attrs { promote: true } }
I20240501 14:04:38.583092 2489 raft_consensus.cc:2932] T c586d994d2414073b25a8b3fab6a7046 P 953c332688d14831b3bc7cfd895e06e9 [term 1 FOLLOWER]: Committing config change with OpId 1.65: config changed from index 47 to 65, 953c332688d14831b3bc7cfd895e06e9 (127.29.124.132) changed from NON_VOTER to VOTER. New config: { opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } } }
I20240501 14:04:38.585417 2622 raft_consensus.cc:2932] T 0f11914838ca4139a1c1813fc5334782 P 4e6923f9753a4450ac4343c34c427d1e [term 1 FOLLOWER]: Committing config change with OpId 1.72: config changed from index 50 to 72, 4e6923f9753a4450ac4343c34c427d1e (127.29.124.133) changed from NON_VOTER to VOTER. New config: { opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: false } } }
I20240501 14:04:38.589944 1050 catalog_manager.cc:5484] T c586d994d2414073b25a8b3fab6a7046 P 953c332688d14831b3bc7cfd895e06e9 reported cstate change: config changed from index 47 to 65, 953c332688d14831b3bc7cfd895e06e9 (127.29.124.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" committed_config { opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } } }
I20240501 14:04:38.593369 1043 catalog_manager.cc:5484] T 0f11914838ca4139a1c1813fc5334782 P 4e6923f9753a4450ac4343c34c427d1e reported cstate change: config changed from index 50 to 72, 4e6923f9753a4450ac4343c34c427d1e (127.29.124.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" committed_config { opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: false } } }
I20240501 14:04:38.617419 1473 tablet_copy_service.cc:140] P eb004c2ac9d244f89b61448584d4115b: Received BeginTabletCopySession request for tablet 6a5325b7aa824831a5fa056fb4439110 from peer 953c332688d14831b3bc7cfd895e06e9 ({username='slave'} at 127.29.124.132:54979)
I20240501 14:04:38.617617 1473 tablet_copy_service.cc:161] P eb004c2ac9d244f89b61448584d4115b: Beginning new tablet copy session on tablet 6a5325b7aa824831a5fa056fb4439110 from peer 953c332688d14831b3bc7cfd895e06e9 at {username='slave'} at 127.29.124.132:54979: session id = 953c332688d14831b3bc7cfd895e06e9-6a5325b7aa824831a5fa056fb4439110
I20240501 14:04:38.620604 1473 tablet_copy_source_session.cc:205] T 6a5325b7aa824831a5fa056fb4439110 P eb004c2ac9d244f89b61448584d4115b: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:04:38.622124 3238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a5325b7aa824831a5fa056fb4439110. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:38.629145 3238 tablet_copy_client.cc:783] T 6a5325b7aa824831a5fa056fb4439110 P 953c332688d14831b3bc7cfd895e06e9: tablet copy: Starting download of 0 data blocks...
I20240501 14:04:38.629526 3238 tablet_copy_client.cc:647] T 6a5325b7aa824831a5fa056fb4439110 P 953c332688d14831b3bc7cfd895e06e9: tablet copy: Starting download of 1 WAL segments...
I20240501 14:04:38.637358 3238 tablet_copy_client.cc:523] T 6a5325b7aa824831a5fa056fb4439110 P 953c332688d14831b3bc7cfd895e06e9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:04:38.640684 3238 tablet_bootstrap.cc:492] T 6a5325b7aa824831a5fa056fb4439110 P 953c332688d14831b3bc7cfd895e06e9: Bootstrap starting.
I20240501 14:04:38.646647 1322 consensus_queue.cc:237] T 53249c71dab94ff4b179309088f4d1cd P 05f4ef3d126d43e9be32731e708ff76c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 79, Committed index: 79, Last appended: 1.79, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: NON_VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: true } }
I20240501 14:04:38.661669 1452 raft_consensus.cc:1261] T c586d994d2414073b25a8b3fab6a7046 P eb004c2ac9d244f89b61448584d4115b [term 1 FOLLOWER]: Refusing update from remote peer 5d45680f41c84ac39bd97c6f00ac2b06: Log matching property violated. Preceding OpId in replica: term: 1 index: 64. Preceding OpId from leader: term: 1 index: 65. (index mismatch)
I20240501 14:04:38.665948 1452 raft_consensus.cc:1261] T 0f11914838ca4139a1c1813fc5334782 P eb004c2ac9d244f89b61448584d4115b [term 1 FOLLOWER]: Refusing update from remote peer 5d45680f41c84ac39bd97c6f00ac2b06: Log matching property violated. Preceding OpId in replica: term: 1 index: 71. Preceding OpId from leader: term: 1 index: 72. (index mismatch)
I20240501 14:04:38.668514 1453 raft_consensus.cc:1261] T 53249c71dab94ff4b179309088f4d1cd P eb004c2ac9d244f89b61448584d4115b [term 1 FOLLOWER]: Refusing update from remote peer 05f4ef3d126d43e9be32731e708ff76c: Log matching property violated. Preceding OpId in replica: term: 1 index: 79. Preceding OpId from leader: term: 1 index: 80. (index mismatch)
I20240501 14:04:38.669085 3046 consensus_queue.cc:1035] T 53249c71dab94ff4b179309088f4d1cd P 05f4ef3d126d43e9be32731e708ff76c [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 80, Last known committed idx: 79, Time since last communication: 0.000s
I20240501 14:04:38.671581 3275 raft_consensus.cc:2932] T 53249c71dab94ff4b179309088f4d1cd P 05f4ef3d126d43e9be32731e708ff76c [term 1 LEADER]: Committing config change with OpId 1.80: config changed from index -1 to 80, NON_VOTER 953c332688d14831b3bc7cfd895e06e9 (127.29.124.132) added. New config: { opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: NON_VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: true } } }
I20240501 14:04:38.672140 1453 raft_consensus.cc:2932] T 53249c71dab94ff4b179309088f4d1cd P eb004c2ac9d244f89b61448584d4115b [term 1 FOLLOWER]: Committing config change with OpId 1.80: config changed from index -1 to 80, NON_VOTER 953c332688d14831b3bc7cfd895e06e9 (127.29.124.132) added. New config: { opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: NON_VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: true } } }
I20240501 14:04:38.678212 1050 catalog_manager.cc:5484] T 53249c71dab94ff4b179309088f4d1cd P eb004c2ac9d244f89b61448584d4115b reported cstate change: config changed from index -1 to 80, NON_VOTER 953c332688d14831b3bc7cfd895e06e9 (127.29.124.132) added. New cstate: current_term: 1 leader_uuid: "05f4ef3d126d43e9be32731e708ff76c" committed_config { opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: NON_VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: true } } }
I20240501 14:04:38.680670 3214 tablet_bootstrap.cc:492] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 752bc4569da647679d5df51641961840: Bootstrap replayed 1/1 log segments. Stats: ops{read=92 overwritten=0 applied=91 ignored=0} inserts{seen=776 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 14:04:38.681082 3214 tablet_bootstrap.cc:492] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 752bc4569da647679d5df51641961840: Bootstrap complete.
I20240501 14:04:38.681387 3214 ts_tablet_manager.cc:1367] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 752bc4569da647679d5df51641961840: Time spent bootstrapping tablet: real 0.743s user 0.144s sys 0.017s
I20240501 14:04:38.685711 3214 raft_consensus.cc:348] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 752bc4569da647679d5df51641961840 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: true } }
I20240501 14:04:38.697512 3214 raft_consensus.cc:729] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 752bc4569da647679d5df51641961840 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 752bc4569da647679d5df51641961840, State: Initialized, Role: LEARNER
I20240501 14:04:38.698096 3214 consensus_queue.cc:260] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 752bc4569da647679d5df51641961840 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 91, Last appended: 1.92, Last appended by leader: 92, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 70 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: true } }
I20240501 14:04:38.699353 3214 ts_tablet_manager.cc:1398] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 752bc4569da647679d5df51641961840: Time spent starting tablet: real 0.018s user 0.001s sys 0.006s
I20240501 14:04:38.707168 1343 tablet_copy_service.cc:342] P 05f4ef3d126d43e9be32731e708ff76c: Request end of tablet copy session 752bc4569da647679d5df51641961840-3a4ca3f4edcd4bf29d0bab40a8856039 received from {username='slave'} at 127.29.124.135:55917
I20240501 14:04:38.707352 1343 tablet_copy_service.cc:433] P 05f4ef3d126d43e9be32731e708ff76c: ending tablet copy session 752bc4569da647679d5df51641961840-3a4ca3f4edcd4bf29d0bab40a8856039 on tablet 3a4ca3f4edcd4bf29d0bab40a8856039 with peer 752bc4569da647679d5df51641961840
I20240501 14:04:38.709981 2885 raft_consensus.cc:1203] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 752bc4569da647679d5df51641961840 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.69->[1.70-1.99] Dedup: 1.92->[1.93-1.99]
I20240501 14:04:38.716728 3319 ts_tablet_manager.cc:902] T 70afcb72943b49a7bea962791b38a91d P 32064197afde4b79bf388b080c365d88: Initiating tablet copy from peer 5d45680f41c84ac39bd97c6f00ac2b06 (127.29.124.129:40925)
W20240501 14:04:38.718293 1258 consensus_peers.cc:485] T 53249c71dab94ff4b179309088f4d1cd P 05f4ef3d126d43e9be32731e708ff76c -> Peer 953c332688d14831b3bc7cfd895e06e9 (127.29.124.132:34305): Couldn't send request to peer 953c332688d14831b3bc7cfd895e06e9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 53249c71dab94ff4b179309088f4d1cd. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:04:38.729859 3065 consensus_queue.cc:1035] T c586d994d2414073b25a8b3fab6a7046 P 5d45680f41c84ac39bd97c6f00ac2b06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 65, Last known committed idx: 63, Time since last communication: 0.000s
I20240501 14:04:38.733510 3153 consensus_queue.cc:1035] T 0f11914838ca4139a1c1813fc5334782 P 5d45680f41c84ac39bd97c6f00ac2b06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 72, Last known committed idx: 70, Time since last communication: 0.000s
I20240501 14:04:38.737007 1452 raft_consensus.cc:2932] T c586d994d2414073b25a8b3fab6a7046 P eb004c2ac9d244f89b61448584d4115b [term 1 FOLLOWER]: Committing config change with OpId 1.65: config changed from index 47 to 65, 953c332688d14831b3bc7cfd895e06e9 (127.29.124.132) changed from NON_VOTER to VOTER. New config: { opid_index: 65 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } } }
I20240501 14:04:38.742400 1192 consensus_queue.cc:237] T 0f11914838ca4139a1c1813fc5334782 P 5d45680f41c84ac39bd97c6f00ac2b06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 79, Committed index: 79, Last appended: 1.79, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: false } }
I20240501 14:04:38.744364 1192 consensus_queue.cc:237] T c586d994d2414073b25a8b3fab6a7046 P 5d45680f41c84ac39bd97c6f00ac2b06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 66, Committed index: 66, Last appended: 1.66, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 67 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } }
I20240501 14:04:38.745828 2489 raft_consensus.cc:1261] T c586d994d2414073b25a8b3fab6a7046 P 953c332688d14831b3bc7cfd895e06e9 [term 1 FOLLOWER]: Refusing update from remote peer 5d45680f41c84ac39bd97c6f00ac2b06: Log matching property violated. Preceding OpId in replica: term: 1 index: 66. Preceding OpId from leader: term: 1 index: 67. (index mismatch)
I20240501 14:04:38.748797 3105 consensus_queue.cc:1035] T c586d994d2414073b25a8b3fab6a7046 P 5d45680f41c84ac39bd97c6f00ac2b06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 67, Last known committed idx: 66, Time since last communication: 0.000s
I20240501 14:04:38.755396 3204 ts_tablet_manager.cc:1398] T 1074c96bc2564edca376f95bd195a62c P 32064197afde4b79bf388b080c365d88: Time spent starting tablet: real 0.239s user 0.039s sys 0.027s
I20240501 14:04:38.760174 3171 raft_consensus.cc:2932] T c586d994d2414073b25a8b3fab6a7046 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 LEADER]: Committing config change with OpId 1.67: config changed from index 65 to 67, VOTER eb004c2ac9d244f89b61448584d4115b (127.29.124.131) evicted. New config: { opid_index: 67 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } } }
I20240501 14:04:38.762082 3319 tablet_copy_client.cc:308] T 70afcb72943b49a7bea962791b38a91d P 32064197afde4b79bf388b080c365d88: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.129:40925
I20240501 14:04:38.763312 1322 consensus_queue.cc:237] T 4c22d4541da544f5b812c6e5e8c2ea40 P 05f4ef3d126d43e9be32731e708ff76c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 95, Committed index: 95, Last appended: 1.95, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } } peers { permanent_uuid: "32064197afde4b79bf388b080c365d88" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 37211 } attrs { promote: true } }
I20240501 14:04:38.766378 2489 raft_consensus.cc:2932] T c586d994d2414073b25a8b3fab6a7046 P 953c332688d14831b3bc7cfd895e06e9 [term 1 FOLLOWER]: Committing config change with OpId 1.67: config changed from index 65 to 67, VOTER eb004c2ac9d244f89b61448584d4115b (127.29.124.131) evicted. New config: { opid_index: 67 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } } }
I20240501 14:04:38.770118 2622 raft_consensus.cc:1261] T 0f11914838ca4139a1c1813fc5334782 P 4e6923f9753a4450ac4343c34c427d1e [term 1 FOLLOWER]: Refusing update from remote peer 5d45680f41c84ac39bd97c6f00ac2b06: Log matching property violated. Preceding OpId in replica: term: 1 index: 79. Preceding OpId from leader: term: 1 index: 80. (index mismatch)
I20240501 14:04:38.765858 1045 catalog_manager.cc:5484] T c586d994d2414073b25a8b3fab6a7046 P 5d45680f41c84ac39bd97c6f00ac2b06 reported cstate change: config changed from index 65 to 67, VOTER eb004c2ac9d244f89b61448584d4115b (127.29.124.131) evicted. New cstate: current_term: 1 leader_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" committed_config { opid_index: 67 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:04:38.771975 2392 consensus_queue.cc:1035] T 0f11914838ca4139a1c1813fc5334782 P 5d45680f41c84ac39bd97c6f00ac2b06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 80, Last known committed idx: 79, Time since last communication: 0.000s
I20240501 14:04:38.772527 1453 raft_consensus.cc:2932] T 0f11914838ca4139a1c1813fc5334782 P eb004c2ac9d244f89b61448584d4115b [term 1 FOLLOWER]: Committing config change with OpId 1.72: config changed from index 50 to 72, 4e6923f9753a4450ac4343c34c427d1e (127.29.124.133) changed from NON_VOTER to VOTER. New config: { opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: false } } }
I20240501 14:04:38.778837 1451 raft_consensus.cc:1261] T 4c22d4541da544f5b812c6e5e8c2ea40 P eb004c2ac9d244f89b61448584d4115b [term 1 FOLLOWER]: Refusing update from remote peer 05f4ef3d126d43e9be32731e708ff76c: Log matching property violated. Preceding OpId in replica: term: 1 index: 95. Preceding OpId from leader: term: 1 index: 96. (index mismatch)
I20240501 14:04:38.780870 3275 consensus_queue.cc:1035] T 4c22d4541da544f5b812c6e5e8c2ea40 P 05f4ef3d126d43e9be32731e708ff76c [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 96, Last known committed idx: 95, Time since last communication: 0.000s
I20240501 14:04:38.781459 1432 tablet_service.cc:1514] Processing DeleteTablet for tablet c586d994d2414073b25a8b3fab6a7046 with delete_type TABLET_DATA_TOMBSTONED (TS eb004c2ac9d244f89b61448584d4115b not found in new config with opid_index 67) from {username='slave'} at 127.0.0.1:38992
I20240501 14:04:38.786059 3277 raft_consensus.cc:2932] T 4c22d4541da544f5b812c6e5e8c2ea40 P 05f4ef3d126d43e9be32731e708ff76c [term 1 LEADER]: Committing config change with OpId 1.96: config changed from index -1 to 96, NON_VOTER 32064197afde4b79bf388b080c365d88 (127.29.124.134) added. New config: { opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } } peers { permanent_uuid: "32064197afde4b79bf388b080c365d88" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 37211 } attrs { promote: true } } }
I20240501 14:04:38.790138 1450 raft_consensus.cc:2932] T 4c22d4541da544f5b812c6e5e8c2ea40 P eb004c2ac9d244f89b61448584d4115b [term 1 FOLLOWER]: Committing config change with OpId 1.96: config changed from index -1 to 96, NON_VOTER 32064197afde4b79bf388b080c365d88 (127.29.124.134) added. New config: { opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } } peers { permanent_uuid: "32064197afde4b79bf388b080c365d88" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 37211 } attrs { promote: true } } }
I20240501 14:04:38.800307 1043 catalog_manager.cc:5484] T 4c22d4541da544f5b812c6e5e8c2ea40 P eb004c2ac9d244f89b61448584d4115b reported cstate change: config changed from index -1 to 96, NON_VOTER 32064197afde4b79bf388b080c365d88 (127.29.124.134) added. New cstate: current_term: 1 leader_uuid: "05f4ef3d126d43e9be32731e708ff76c" committed_config { opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } } peers { permanent_uuid: "32064197afde4b79bf388b080c365d88" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 37211 } attrs { promote: true } } }
I20240501 14:04:38.801105 3065 raft_consensus.cc:2932] T 0f11914838ca4139a1c1813fc5334782 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 LEADER]: Committing config change with OpId 1.80: config changed from index 72 to 80, VOTER eb004c2ac9d244f89b61448584d4115b (127.29.124.131) evicted. New config: { opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: false } } }
I20240501 14:04:38.806468 2622 raft_consensus.cc:2932] T 0f11914838ca4139a1c1813fc5334782 P 4e6923f9753a4450ac4343c34c427d1e [term 1 FOLLOWER]: Committing config change with OpId 1.80: config changed from index 72 to 80, VOTER eb004c2ac9d244f89b61448584d4115b (127.29.124.131) evicted. New config: { opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: false } } }
I20240501 14:04:38.807850 1050 catalog_manager.cc:5484] T 0f11914838ca4139a1c1813fc5334782 P 5d45680f41c84ac39bd97c6f00ac2b06 reported cstate change: config changed from index 72 to 80, VOTER eb004c2ac9d244f89b61448584d4115b (127.29.124.131) evicted. New cstate: current_term: 1 leader_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" committed_config { opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:04:38.812561 1433 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f11914838ca4139a1c1813fc5334782 with delete_type TABLET_DATA_TOMBSTONED (TS eb004c2ac9d244f89b61448584d4115b not found in new config with opid_index 80) from {username='slave'} at 127.0.0.1:38992
I20240501 14:04:38.823096 1036 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c586d994d2414073b25a8b3fab6a7046 with cas_config_opid_index 65: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:04:38.823381 1036 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0f11914838ca4139a1c1813fc5334782 with cas_config_opid_index 72: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:04:38.825740 1213 tablet_copy_service.cc:342] P 5d45680f41c84ac39bd97c6f00ac2b06: Request end of tablet copy session 32064197afde4b79bf388b080c365d88-1074c96bc2564edca376f95bd195a62c received from {username='slave'} at 127.29.124.134:41391
I20240501 14:04:38.825882 1213 tablet_copy_service.cc:433] P 5d45680f41c84ac39bd97c6f00ac2b06: ending tablet copy session 32064197afde4b79bf388b080c365d88-1074c96bc2564edca376f95bd195a62c on tablet 1074c96bc2564edca376f95bd195a62c with peer 32064197afde4b79bf388b080c365d88
I20240501 14:04:38.826742 1213 tablet_copy_service.cc:140] P 5d45680f41c84ac39bd97c6f00ac2b06: Received BeginTabletCopySession request for tablet 70afcb72943b49a7bea962791b38a91d from peer 32064197afde4b79bf388b080c365d88 ({username='slave'} at 127.29.124.134:41391)
I20240501 14:04:38.826903 1213 tablet_copy_service.cc:161] P 5d45680f41c84ac39bd97c6f00ac2b06: Beginning new tablet copy session on tablet 70afcb72943b49a7bea962791b38a91d from peer 32064197afde4b79bf388b080c365d88 at {username='slave'} at 127.29.124.134:41391: session id = 32064197afde4b79bf388b080c365d88-70afcb72943b49a7bea962791b38a91d
I20240501 14:04:38.830237 1213 tablet_copy_source_session.cc:205] T 70afcb72943b49a7bea962791b38a91d P 5d45680f41c84ac39bd97c6f00ac2b06: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:04:38.845542 3330 tablet_replica.cc:330] T c586d994d2414073b25a8b3fab6a7046 P eb004c2ac9d244f89b61448584d4115b: stopping tablet replica
I20240501 14:04:38.847573 3330 raft_consensus.cc:2229] T c586d994d2414073b25a8b3fab6a7046 P eb004c2ac9d244f89b61448584d4115b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:04:38.850766 3330 raft_consensus.cc:2258] T c586d994d2414073b25a8b3fab6a7046 P eb004c2ac9d244f89b61448584d4115b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:04:38.852952 3319 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 70afcb72943b49a7bea962791b38a91d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:38.858891 3330 ts_tablet_manager.cc:1875] T c586d994d2414073b25a8b3fab6a7046 P eb004c2ac9d244f89b61448584d4115b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:04:38.862310 3319 tablet_copy_client.cc:783] T 70afcb72943b49a7bea962791b38a91d P 32064197afde4b79bf388b080c365d88: tablet copy: Starting download of 0 data blocks...
I20240501 14:04:38.862649 3319 tablet_copy_client.cc:647] T 70afcb72943b49a7bea962791b38a91d P 32064197afde4b79bf388b080c365d88: tablet copy: Starting download of 1 WAL segments...
I20240501 14:04:38.878588 3319 tablet_copy_client.cc:523] T 70afcb72943b49a7bea962791b38a91d P 32064197afde4b79bf388b080c365d88: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:04:38.885697 3319 tablet_bootstrap.cc:492] T 70afcb72943b49a7bea962791b38a91d P 32064197afde4b79bf388b080c365d88: Bootstrap starting.
W20240501 14:04:38.891939 1256 consensus_peers.cc:485] T 4c22d4541da544f5b812c6e5e8c2ea40 P 05f4ef3d126d43e9be32731e708ff76c -> Peer 32064197afde4b79bf388b080c365d88 (127.29.124.134:37211): Couldn't send request to peer 32064197afde4b79bf388b080c365d88. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4c22d4541da544f5b812c6e5e8c2ea40. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:04:38.892637 3330 ts_tablet_manager.cc:1888] T c586d994d2414073b25a8b3fab6a7046 P eb004c2ac9d244f89b61448584d4115b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.66
I20240501 14:04:38.906064 3330 log.cc:1196] T c586d994d2414073b25a8b3fab6a7046 P eb004c2ac9d244f89b61448584d4115b: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/wal/wals/c586d994d2414073b25a8b3fab6a7046
I20240501 14:04:38.907963 1039 catalog_manager.cc:4830] TS eb004c2ac9d244f89b61448584d4115b (127.29.124.131:40225): tablet c586d994d2414073b25a8b3fab6a7046 (table rebalance_test_table_4 [id=6fd9bb23d84f4078a694c2c7a0cb93b2]) successfully deleted
I20240501 14:04:38.908365 3330 tablet_replica.cc:330] T 0f11914838ca4139a1c1813fc5334782 P eb004c2ac9d244f89b61448584d4115b: stopping tablet replica
I20240501 14:04:38.908576 3330 raft_consensus.cc:2229] T 0f11914838ca4139a1c1813fc5334782 P eb004c2ac9d244f89b61448584d4115b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:04:38.909489 3330 raft_consensus.cc:2258] T 0f11914838ca4139a1c1813fc5334782 P eb004c2ac9d244f89b61448584d4115b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:04:38.910470 3330 ts_tablet_manager.cc:1875] T 0f11914838ca4139a1c1813fc5334782 P eb004c2ac9d244f89b61448584d4115b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:04:38.930016 3277 raft_consensus.cc:1050] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 05f4ef3d126d43e9be32731e708ff76c: attempting to promote NON_VOTER 752bc4569da647679d5df51641961840 to VOTER
I20240501 14:04:38.930938 3277 consensus_queue.cc:237] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 05f4ef3d126d43e9be32731e708ff76c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 101, Committed index: 101, Last appended: 1.102, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false } }
I20240501 14:04:38.932988 2885 raft_consensus.cc:1261] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 752bc4569da647679d5df51641961840 [term 1 LEARNER]: Refusing update from remote peer 05f4ef3d126d43e9be32731e708ff76c: Log matching property violated. Preceding OpId in replica: term: 1 index: 102. Preceding OpId from leader: term: 1 index: 103. (index mismatch)
I20240501 14:04:38.946609 2989 consensus_queue.cc:1035] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 05f4ef3d126d43e9be32731e708ff76c [LEADER]: Connected to new peer: Peer: permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 103, Last known committed idx: 101, Time since last communication: 0.000s
I20240501 14:04:38.952946 2287 raft_consensus.cc:2932] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 05f4ef3d126d43e9be32731e708ff76c [term 1 LEADER]: Committing config change with OpId 1.103: config changed from index 70 to 103, 752bc4569da647679d5df51641961840 (127.29.124.135) changed from NON_VOTER to VOTER. New config: { opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false } } }
I20240501 14:04:38.955708 3330 ts_tablet_manager.cc:1888] T 0f11914838ca4139a1c1813fc5334782 P eb004c2ac9d244f89b61448584d4115b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.79
I20240501 14:04:38.955832 3330 log.cc:1196] T 0f11914838ca4139a1c1813fc5334782 P eb004c2ac9d244f89b61448584d4115b: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/wal/wals/0f11914838ca4139a1c1813fc5334782
I20240501 14:04:38.956518 1039 catalog_manager.cc:4830] TS eb004c2ac9d244f89b61448584d4115b (127.29.124.131:40225): tablet 0f11914838ca4139a1c1813fc5334782 (table rebalance_test_table_3 [id=16c6cbe8fef84c88b36c2d1809ebe48b]) successfully deleted
I20240501 14:04:38.958057 2885 raft_consensus.cc:2932] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 752bc4569da647679d5df51641961840 [term 1 FOLLOWER]: Committing config change with OpId 1.103: config changed from index 70 to 103, 752bc4569da647679d5df51641961840 (127.29.124.135) changed from NON_VOTER to VOTER. New config: { opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false } } }
I20240501 14:04:38.958647 1045 catalog_manager.cc:5484] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 05f4ef3d126d43e9be32731e708ff76c reported cstate change: config changed from index 70 to 103, 752bc4569da647679d5df51641961840 (127.29.124.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "05f4ef3d126d43e9be32731e708ff76c" committed_config { opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:04:38.962747 1322 consensus_queue.cc:237] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 05f4ef3d126d43e9be32731e708ff76c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 103, Committed index: 103, Last appended: 1.103, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false } }
I20240501 14:04:38.965085 2884 raft_consensus.cc:1261] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 752bc4569da647679d5df51641961840 [term 1 FOLLOWER]: Refusing update from remote peer 05f4ef3d126d43e9be32731e708ff76c: Log matching property violated. Preceding OpId in replica: term: 1 index: 103. Preceding OpId from leader: term: 1 index: 104. (index mismatch)
I20240501 14:04:38.965705 2989 consensus_queue.cc:1035] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 05f4ef3d126d43e9be32731e708ff76c [LEADER]: Connected to new peer: Peer: permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 104, Last known committed idx: 103, Time since last communication: 0.000s
I20240501 14:04:38.970703 3046 raft_consensus.cc:2932] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 05f4ef3d126d43e9be32731e708ff76c [term 1 LEADER]: Committing config change with OpId 1.104: config changed from index 103 to 104, VOTER 5d45680f41c84ac39bd97c6f00ac2b06 (127.29.124.129) evicted. New config: { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false } } }
I20240501 14:04:38.976564 1038 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3a4ca3f4edcd4bf29d0bab40a8856039 with cas_config_opid_index 103: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:04:38.977077 2885 raft_consensus.cc:2932] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 752bc4569da647679d5df51641961840 [term 1 FOLLOWER]: Committing config change with OpId 1.104: config changed from index 103 to 104, VOTER 5d45680f41c84ac39bd97c6f00ac2b06 (127.29.124.129) evicted. New config: { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false } } }
I20240501 14:04:38.977666 1045 catalog_manager.cc:5484] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 05f4ef3d126d43e9be32731e708ff76c reported cstate change: config changed from index 103 to 104, VOTER 5d45680f41c84ac39bd97c6f00ac2b06 (127.29.124.129) evicted. New cstate: current_term: 1 leader_uuid: "05f4ef3d126d43e9be32731e708ff76c" committed_config { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:04:38.981096 1192 raft_consensus.cc:1261] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 FOLLOWER]: Refusing update from remote peer 05f4ef3d126d43e9be32731e708ff76c: Log matching property violated. Preceding OpId in replica: term: 1 index: 101. Preceding OpId from leader: term: 1 index: 103. (index mismatch)
I20240501 14:04:39.006430 1172 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a4ca3f4edcd4bf29d0bab40a8856039 with delete_type TABLET_DATA_TOMBSTONED (TS 5d45680f41c84ac39bd97c6f00ac2b06 not found in new config with opid_index 104) from {username='slave'} at 127.0.0.1:51144
I20240501 14:04:39.013566 3214 ts_tablet_manager.cc:902] T db2b3f73d20c4790808bbe0f1d5e489c P 752bc4569da647679d5df51641961840: Initiating tablet copy from peer 05f4ef3d126d43e9be32731e708ff76c (127.29.124.130:41511)
I20240501 14:04:39.020641 3341 tablet_replica.cc:330] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 5d45680f41c84ac39bd97c6f00ac2b06: stopping tablet replica
I20240501 14:04:39.021128 3341 raft_consensus.cc:2229] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:04:39.050436 3341 raft_consensus.cc:2258] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:04:39.062845 3341 ts_tablet_manager.cc:1875] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 5d45680f41c84ac39bd97c6f00ac2b06: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:04:39.071403 2753 raft_consensus.cc:1203] T 1074c96bc2564edca376f95bd195a62c P 32064197afde4b79bf388b080c365d88 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.62->[1.63-1.105] Dedup: 1.81->[1.82-1.105]
I20240501 14:04:39.073139 3341 ts_tablet_manager.cc:1888] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 5d45680f41c84ac39bd97c6f00ac2b06: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.101
I20240501 14:04:39.080327 3214 tablet_copy_client.cc:308] T db2b3f73d20c4790808bbe0f1d5e489c P 752bc4569da647679d5df51641961840: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.130:41511
I20240501 14:04:39.081009 1343 tablet_copy_service.cc:140] P 05f4ef3d126d43e9be32731e708ff76c: Received BeginTabletCopySession request for tablet db2b3f73d20c4790808bbe0f1d5e489c from peer 752bc4569da647679d5df51641961840 ({username='slave'} at 127.29.124.135:55917)
I20240501 14:04:39.081168 1343 tablet_copy_service.cc:161] P 05f4ef3d126d43e9be32731e708ff76c: Beginning new tablet copy session on tablet db2b3f73d20c4790808bbe0f1d5e489c from peer 752bc4569da647679d5df51641961840 at {username='slave'} at 127.29.124.135:55917: session id = 752bc4569da647679d5df51641961840-db2b3f73d20c4790808bbe0f1d5e489c
I20240501 14:04:39.084374 1343 tablet_copy_source_session.cc:205] T db2b3f73d20c4790808bbe0f1d5e489c P 05f4ef3d126d43e9be32731e708ff76c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:04:39.085713 3214 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db2b3f73d20c4790808bbe0f1d5e489c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:39.100337 3214 tablet_copy_client.cc:783] T db2b3f73d20c4790808bbe0f1d5e489c P 752bc4569da647679d5df51641961840: tablet copy: Starting download of 0 data blocks...
I20240501 14:04:39.100778 3214 tablet_copy_client.cc:647] T db2b3f73d20c4790808bbe0f1d5e489c P 752bc4569da647679d5df51641961840: tablet copy: Starting download of 1 WAL segments...
I20240501 14:04:39.106465 3214 tablet_copy_client.cc:523] T db2b3f73d20c4790808bbe0f1d5e489c P 752bc4569da647679d5df51641961840: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:04:39.110040 3341 log.cc:1196] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 5d45680f41c84ac39bd97c6f00ac2b06: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/wal/wals/3a4ca3f4edcd4bf29d0bab40a8856039
I20240501 14:04:39.114885 3354 ts_tablet_manager.cc:902] T 53249c71dab94ff4b179309088f4d1cd P 953c332688d14831b3bc7cfd895e06e9: Initiating tablet copy from peer 05f4ef3d126d43e9be32731e708ff76c (127.29.124.130:41511)
I20240501 14:04:39.125685 3214 tablet_bootstrap.cc:492] T db2b3f73d20c4790808bbe0f1d5e489c P 752bc4569da647679d5df51641961840: Bootstrap starting.
I20240501 14:04:39.134106 3249 tablet_bootstrap.cc:492] T 14e026c65736401fb3aec7e6462ae001 P 32064197afde4b79bf388b080c365d88: Bootstrap replayed 1/1 log segments. Stats: ops{read=88 overwritten=0 applied=88 ignored=0} inserts{seen=719 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:04:39.138803 3249 tablet_bootstrap.cc:492] T 14e026c65736401fb3aec7e6462ae001 P 32064197afde4b79bf388b080c365d88: Bootstrap complete.
I20240501 14:04:39.139230 3249 ts_tablet_manager.cc:1367] T 14e026c65736401fb3aec7e6462ae001 P 32064197afde4b79bf388b080c365d88: Time spent bootstrapping tablet: real 0.958s user 0.142s sys 0.025s
I20240501 14:04:39.140277 3249 raft_consensus.cc:348] T 14e026c65736401fb3aec7e6462ae001 P 32064197afde4b79bf388b080c365d88 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "32064197afde4b79bf388b080c365d88" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 37211 } attrs { promote: true } }
I20240501 14:04:39.140969 3249 raft_consensus.cc:729] T 14e026c65736401fb3aec7e6462ae001 P 32064197afde4b79bf388b080c365d88 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 32064197afde4b79bf388b080c365d88, State: Initialized, Role: LEARNER
I20240501 14:04:39.141230 3249 consensus_queue.cc:260] T 14e026c65736401fb3aec7e6462ae001 P 32064197afde4b79bf388b080c365d88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 88, Last appended: 1.88, Last appended by leader: 88, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 77 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "32064197afde4b79bf388b080c365d88" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 37211 } attrs { promote: true } }
I20240501 14:04:39.141968 3249 ts_tablet_manager.cc:1398] T 14e026c65736401fb3aec7e6462ae001 P 32064197afde4b79bf388b080c365d88: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:04:39.142714 1473 tablet_copy_service.cc:342] P eb004c2ac9d244f89b61448584d4115b: Request end of tablet copy session 32064197afde4b79bf388b080c365d88-14e026c65736401fb3aec7e6462ae001 received from {username='slave'} at 127.29.124.134:43151
I20240501 14:04:39.142860 1473 tablet_copy_service.cc:433] P eb004c2ac9d244f89b61448584d4115b: ending tablet copy session 32064197afde4b79bf388b080c365d88-14e026c65736401fb3aec7e6462ae001 on tablet 14e026c65736401fb3aec7e6462ae001 with peer 32064197afde4b79bf388b080c365d88
I20240501 14:04:39.148958 1036 catalog_manager.cc:4830] TS 5d45680f41c84ac39bd97c6f00ac2b06 (127.29.124.129:40925): tablet 3a4ca3f4edcd4bf29d0bab40a8856039 (table rebalance_test_table_2 [id=9c1d190b2b264f27af23180ab8f30d42]) successfully deleted
I20240501 14:04:39.156467 2752 raft_consensus.cc:1203] T 14e026c65736401fb3aec7e6462ae001 P 32064197afde4b79bf388b080c365d88 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.76->[1.77-1.98] Dedup: 1.88->[1.89-1.98]
I20240501 14:04:39.187981 3171 raft_consensus.cc:1050] T 1074c96bc2564edca376f95bd195a62c P 5d45680f41c84ac39bd97c6f00ac2b06: attempting to promote NON_VOTER 32064197afde4b79bf388b080c365d88 to VOTER
I20240501 14:04:39.188732 3171 consensus_queue.cc:237] T 1074c96bc2564edca376f95bd195a62c P 5d45680f41c84ac39bd97c6f00ac2b06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 110, Committed index: 110, Last appended: 1.110, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 111 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "32064197afde4b79bf388b080c365d88" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 37211 } attrs { promote: false } }
I20240501 14:04:39.195546 2753 raft_consensus.cc:1261] T 1074c96bc2564edca376f95bd195a62c P 32064197afde4b79bf388b080c365d88 [term 1 LEARNER]: Refusing update from remote peer 5d45680f41c84ac39bd97c6f00ac2b06: Log matching property violated. Preceding OpId in replica: term: 1 index: 110. Preceding OpId from leader: term: 1 index: 111. (index mismatch)
I20240501 14:04:39.195955 1323 raft_consensus.cc:1261] T 1074c96bc2564edca376f95bd195a62c P 05f4ef3d126d43e9be32731e708ff76c [term 1 FOLLOWER]: Refusing update from remote peer 5d45680f41c84ac39bd97c6f00ac2b06: Log matching property violated. Preceding OpId in replica: term: 1 index: 110. Preceding OpId from leader: term: 1 index: 111. (index mismatch)
I20240501 14:04:39.196903 3153 consensus_queue.cc:1035] T 1074c96bc2564edca376f95bd195a62c P 5d45680f41c84ac39bd97c6f00ac2b06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32064197afde4b79bf388b080c365d88" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 37211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 111, Last known committed idx: 110, Time since last communication: 0.000s
I20240501 14:04:39.197362 3153 consensus_queue.cc:1035] T 1074c96bc2564edca376f95bd195a62c P 5d45680f41c84ac39bd97c6f00ac2b06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 111, Last known committed idx: 110, Time since last communication: 0.000s
I20240501 14:04:39.200492 3066 raft_consensus.cc:2932] T 1074c96bc2564edca376f95bd195a62c P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 LEADER]: Committing config change with OpId 1.111: config changed from index 63 to 111, 32064197afde4b79bf388b080c365d88 (127.29.124.134) changed from NON_VOTER to VOTER. New config: { opid_index: 111 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "32064197afde4b79bf388b080c365d88" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 37211 } attrs { promote: false } } }
I20240501 14:04:39.200683 1323 raft_consensus.cc:2932] T 1074c96bc2564edca376f95bd195a62c P 05f4ef3d126d43e9be32731e708ff76c [term 1 FOLLOWER]: Committing config change with OpId 1.111: config changed from index 63 to 111, 32064197afde4b79bf388b080c365d88 (127.29.124.134) changed from NON_VOTER to VOTER. New config: { opid_index: 111 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "32064197afde4b79bf388b080c365d88" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 37211 } attrs { promote: false } } }
I20240501 14:04:39.206308 1045 catalog_manager.cc:5484] T 1074c96bc2564edca376f95bd195a62c P 5d45680f41c84ac39bd97c6f00ac2b06 reported cstate change: config changed from index 63 to 111, 32064197afde4b79bf388b080c365d88 (127.29.124.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" committed_config { opid_index: 111 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32064197afde4b79bf388b080c365d88" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 37211 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:04:39.209290 2753 raft_consensus.cc:2932] T 1074c96bc2564edca376f95bd195a62c P 32064197afde4b79bf388b080c365d88 [term 1 FOLLOWER]: Committing config change with OpId 1.111: config changed from index 63 to 111, 32064197afde4b79bf388b080c365d88 (127.29.124.134) changed from NON_VOTER to VOTER. New config: { opid_index: 111 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "32064197afde4b79bf388b080c365d88" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 37211 } attrs { promote: false } } }
W20240501 14:04:39.249462 1129 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.124.131:40225, user_credentials={real_user=slave}} blocked reactor thread for 73941.2us
I20240501 14:04:39.254745 3201 raft_consensus.cc:1050] T 14e026c65736401fb3aec7e6462ae001 P eb004c2ac9d244f89b61448584d4115b: attempting to promote NON_VOTER 32064197afde4b79bf388b080c365d88 to VOTER
I20240501 14:04:39.255679 3201 consensus_queue.cc:237] T 14e026c65736401fb3aec7e6462ae001 P eb004c2ac9d244f89b61448584d4115b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 98, Committed index: 98, Last appended: 1.98, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "32064197afde4b79bf388b080c365d88" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 37211 } attrs { promote: false } }
I20240501 14:04:39.258541 2752 raft_consensus.cc:1261] T 14e026c65736401fb3aec7e6462ae001 P 32064197afde4b79bf388b080c365d88 [term 1 LEARNER]: Refusing update from remote peer eb004c2ac9d244f89b61448584d4115b: Log matching property violated. Preceding OpId in replica: term: 1 index: 98. Preceding OpId from leader: term: 1 index: 99. (index mismatch)
I20240501 14:04:39.258560 1193 raft_consensus.cc:1261] T 14e026c65736401fb3aec7e6462ae001 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 FOLLOWER]: Refusing update from remote peer eb004c2ac9d244f89b61448584d4115b: Log matching property violated. Preceding OpId in replica: term: 1 index: 98. Preceding OpId from leader: term: 1 index: 99. (index mismatch)
I20240501 14:04:39.259141 2970 consensus_queue.cc:1035] T 14e026c65736401fb3aec7e6462ae001 P eb004c2ac9d244f89b61448584d4115b [LEADER]: Connected to new peer: Peer: permanent_uuid: "32064197afde4b79bf388b080c365d88" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 37211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 99, Last known committed idx: 98, Time since last communication: 0.000s
I20240501 14:04:39.260282 2970 consensus_queue.cc:1035] T 14e026c65736401fb3aec7e6462ae001 P eb004c2ac9d244f89b61448584d4115b [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 99, Last known committed idx: 98, Time since last communication: 0.000s
I20240501 14:04:39.262073 3261 raft_consensus.cc:2932] T 14e026c65736401fb3aec7e6462ae001 P eb004c2ac9d244f89b61448584d4115b [term 1 LEADER]: Committing config change with OpId 1.99: config changed from index 77 to 99, 32064197afde4b79bf388b080c365d88 (127.29.124.134) changed from NON_VOTER to VOTER. New config: { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "32064197afde4b79bf388b080c365d88" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 37211 } attrs { promote: false } } }
I20240501 14:04:39.262646 2752 raft_consensus.cc:2932] T 14e026c65736401fb3aec7e6462ae001 P 32064197afde4b79bf388b080c365d88 [term 1 FOLLOWER]: Committing config change with OpId 1.99: config changed from index 77 to 99, 32064197afde4b79bf388b080c365d88 (127.29.124.134) changed from NON_VOTER to VOTER. New config: { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "32064197afde4b79bf388b080c365d88" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 37211 } attrs { promote: false } } }
I20240501 14:04:39.263581 1193 raft_consensus.cc:2932] T 14e026c65736401fb3aec7e6462ae001 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 FOLLOWER]: Committing config change with OpId 1.99: config changed from index 77 to 99, 32064197afde4b79bf388b080c365d88 (127.29.124.134) changed from NON_VOTER to VOTER. New config: { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "32064197afde4b79bf388b080c365d88" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 37211 } attrs { promote: false } } }
I20240501 14:04:39.267287 1192 consensus_queue.cc:237] T 1074c96bc2564edca376f95bd195a62c P 5d45680f41c84ac39bd97c6f00ac2b06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 112, Committed index: 112, Last appended: 1.112, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 113 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "32064197afde4b79bf388b080c365d88" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 37211 } attrs { promote: false } }
I20240501 14:04:39.268415 1043 catalog_manager.cc:5484] T 14e026c65736401fb3aec7e6462ae001 P eb004c2ac9d244f89b61448584d4115b reported cstate change: config changed from index 77 to 99, 32064197afde4b79bf388b080c365d88 (127.29.124.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "eb004c2ac9d244f89b61448584d4115b" committed_config { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32064197afde4b79bf388b080c365d88" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 37211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:04:39.269922 2753 raft_consensus.cc:1261] T 1074c96bc2564edca376f95bd195a62c P 32064197afde4b79bf388b080c365d88 [term 1 FOLLOWER]: Refusing update from remote peer 5d45680f41c84ac39bd97c6f00ac2b06: Log matching property violated. Preceding OpId in replica: term: 1 index: 112. Preceding OpId from leader: term: 1 index: 113. (index mismatch)
I20240501 14:04:39.273515 3171 consensus_queue.cc:1035] T 1074c96bc2564edca376f95bd195a62c P 5d45680f41c84ac39bd97c6f00ac2b06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32064197afde4b79bf388b080c365d88" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 37211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 113, Last known committed idx: 112, Time since last communication: 0.003s
I20240501 14:04:39.277767 1453 consensus_queue.cc:237] T 14e026c65736401fb3aec7e6462ae001 P eb004c2ac9d244f89b61448584d4115b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 99, Committed index: 99, Last appended: 1.99, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 100 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "32064197afde4b79bf388b080c365d88" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 37211 } attrs { promote: false } }
I20240501 14:04:39.279372 3170 raft_consensus.cc:2932] T 1074c96bc2564edca376f95bd195a62c P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 LEADER]: Committing config change with OpId 1.113: config changed from index 111 to 113, VOTER 05f4ef3d126d43e9be32731e708ff76c (127.29.124.130) evicted. New config: { opid_index: 113 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "32064197afde4b79bf388b080c365d88" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 37211 } attrs { promote: false } } }
I20240501 14:04:39.280346 2752 raft_consensus.cc:1261] T 14e026c65736401fb3aec7e6462ae001 P 32064197afde4b79bf388b080c365d88 [term 1 FOLLOWER]: Refusing update from remote peer eb004c2ac9d244f89b61448584d4115b: Log matching property violated. Preceding OpId in replica: term: 1 index: 99. Preceding OpId from leader: term: 1 index: 100. (index mismatch)
I20240501 14:04:39.281019 3201 consensus_queue.cc:1035] T 14e026c65736401fb3aec7e6462ae001 P eb004c2ac9d244f89b61448584d4115b [LEADER]: Connected to new peer: Peer: permanent_uuid: "32064197afde4b79bf388b080c365d88" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 37211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 100, Last known committed idx: 99, Time since last communication: 0.000s
I20240501 14:04:39.284150 3013 raft_consensus.cc:2932] T 14e026c65736401fb3aec7e6462ae001 P eb004c2ac9d244f89b61448584d4115b [term 1 LEADER]: Committing config change with OpId 1.100: config changed from index 99 to 100, VOTER 5d45680f41c84ac39bd97c6f00ac2b06 (127.29.124.129) evicted. New config: { opid_index: 100 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "32064197afde4b79bf388b080c365d88" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 37211 } attrs { promote: false } } }
I20240501 14:04:39.283864 2752 raft_consensus.cc:2932] T 1074c96bc2564edca376f95bd195a62c P 32064197afde4b79bf388b080c365d88 [term 1 FOLLOWER]: Committing config change with OpId 1.113: config changed from index 111 to 113, VOTER 05f4ef3d126d43e9be32731e708ff76c (127.29.124.130) evicted. New config: { opid_index: 113 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "32064197afde4b79bf388b080c365d88" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 37211 } attrs { promote: false } } }
I20240501 14:04:39.287670 1036 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1074c96bc2564edca376f95bd195a62c with cas_config_opid_index 111: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:04:39.289443 1039 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 14e026c65736401fb3aec7e6462ae001 with cas_config_opid_index 99: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:04:39.292629 1050 catalog_manager.cc:5484] T 1074c96bc2564edca376f95bd195a62c P 32064197afde4b79bf388b080c365d88 reported cstate change: config changed from index 111 to 113, VOTER 05f4ef3d126d43e9be32731e708ff76c (127.29.124.130) evicted. New cstate: current_term: 1 leader_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" committed_config { opid_index: 113 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "32064197afde4b79bf388b080c365d88" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 37211 } attrs { promote: false } } }
I20240501 14:04:39.295608 3249 ts_tablet_manager.cc:902] T 4c22d4541da544f5b812c6e5e8c2ea40 P 32064197afde4b79bf388b080c365d88: Initiating tablet copy from peer 05f4ef3d126d43e9be32731e708ff76c (127.29.124.130:41511)
I20240501 14:04:39.297550 2753 raft_consensus.cc:2932] T 14e026c65736401fb3aec7e6462ae001 P 32064197afde4b79bf388b080c365d88 [term 1 FOLLOWER]: Committing config change with OpId 1.100: config changed from index 99 to 100, VOTER 5d45680f41c84ac39bd97c6f00ac2b06 (127.29.124.129) evicted. New config: { opid_index: 100 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "32064197afde4b79bf388b080c365d88" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 37211 } attrs { promote: false } } }
I20240501 14:04:39.302976 1302 tablet_service.cc:1514] Processing DeleteTablet for tablet 1074c96bc2564edca376f95bd195a62c with delete_type TABLET_DATA_TOMBSTONED (TS 05f4ef3d126d43e9be32731e708ff76c not found in new config with opid_index 113) from {username='slave'} at 127.0.0.1:35916
I20240501 14:04:39.305418 1043 catalog_manager.cc:5484] T 14e026c65736401fb3aec7e6462ae001 P eb004c2ac9d244f89b61448584d4115b reported cstate change: config changed from index 99 to 100, VOTER 5d45680f41c84ac39bd97c6f00ac2b06 (127.29.124.129) evicted. New cstate: current_term: 1 leader_uuid: "eb004c2ac9d244f89b61448584d4115b" committed_config { opid_index: 100 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32064197afde4b79bf388b080c365d88" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 37211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:04:39.324019 3360 tablet_replica.cc:330] T 1074c96bc2564edca376f95bd195a62c P 05f4ef3d126d43e9be32731e708ff76c: stopping tablet replica
I20240501 14:04:39.324394 1172 tablet_service.cc:1514] Processing DeleteTablet for tablet 14e026c65736401fb3aec7e6462ae001 with delete_type TABLET_DATA_TOMBSTONED (TS 5d45680f41c84ac39bd97c6f00ac2b06 not found in new config with opid_index 100) from {username='slave'} at 127.0.0.1:51144
I20240501 14:04:39.324884 3341 tablet_replica.cc:330] T 14e026c65736401fb3aec7e6462ae001 P 5d45680f41c84ac39bd97c6f00ac2b06: stopping tablet replica
I20240501 14:04:39.325101 3341 raft_consensus.cc:2229] T 14e026c65736401fb3aec7e6462ae001 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:04:39.325770 3341 raft_consensus.cc:2258] T 14e026c65736401fb3aec7e6462ae001 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:04:39.327250 3360 raft_consensus.cc:2229] T 1074c96bc2564edca376f95bd195a62c P 05f4ef3d126d43e9be32731e708ff76c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:04:39.328428 3360 raft_consensus.cc:2258] T 1074c96bc2564edca376f95bd195a62c P 05f4ef3d126d43e9be32731e708ff76c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:04:39.329624 3341 ts_tablet_manager.cc:1875] T 14e026c65736401fb3aec7e6462ae001 P 5d45680f41c84ac39bd97c6f00ac2b06: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:04:39.331056 3360 ts_tablet_manager.cc:1875] T 1074c96bc2564edca376f95bd195a62c P 05f4ef3d126d43e9be32731e708ff76c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:04:39.334833 3341 ts_tablet_manager.cc:1888] T 14e026c65736401fb3aec7e6462ae001 P 5d45680f41c84ac39bd97c6f00ac2b06: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.99
I20240501 14:04:39.335027 3341 log.cc:1196] T 14e026c65736401fb3aec7e6462ae001 P 5d45680f41c84ac39bd97c6f00ac2b06: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/wal/wals/14e026c65736401fb3aec7e6462ae001
I20240501 14:04:39.336454 1036 catalog_manager.cc:4830] TS 5d45680f41c84ac39bd97c6f00ac2b06 (127.29.124.129:40925): tablet 14e026c65736401fb3aec7e6462ae001 (table rebalance_test_table_1 [id=423117c09096433e93fec4726745a428]) successfully deleted
I20240501 14:04:39.367741 3360 ts_tablet_manager.cc:1888] T 1074c96bc2564edca376f95bd195a62c P 05f4ef3d126d43e9be32731e708ff76c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.112
I20240501 14:04:39.368903 3360 log.cc:1196] T 1074c96bc2564edca376f95bd195a62c P 05f4ef3d126d43e9be32731e708ff76c: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/wal/wals/1074c96bc2564edca376f95bd195a62c
I20240501 14:04:39.370800 1038 catalog_manager.cc:4830] TS 05f4ef3d126d43e9be32731e708ff76c (127.29.124.130:41511): tablet 1074c96bc2564edca376f95bd195a62c (table rebalance_test_table_3 [id=16c6cbe8fef84c88b36c2d1809ebe48b]) successfully deleted
I20240501 14:04:39.427332 3354 tablet_copy_client.cc:308] T 53249c71dab94ff4b179309088f4d1cd P 953c332688d14831b3bc7cfd895e06e9: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.130:41511
I20240501 14:04:39.434796 3249 tablet_copy_client.cc:308] T 4c22d4541da544f5b812c6e5e8c2ea40 P 32064197afde4b79bf388b080c365d88: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.130:41511
I20240501 14:04:39.492904 1343 tablet_copy_service.cc:140] P 05f4ef3d126d43e9be32731e708ff76c: Received BeginTabletCopySession request for tablet 53249c71dab94ff4b179309088f4d1cd from peer 953c332688d14831b3bc7cfd895e06e9 ({username='slave'} at 127.29.124.132:38673)
I20240501 14:04:39.493070 1343 tablet_copy_service.cc:161] P 05f4ef3d126d43e9be32731e708ff76c: Beginning new tablet copy session on tablet 53249c71dab94ff4b179309088f4d1cd from peer 953c332688d14831b3bc7cfd895e06e9 at {username='slave'} at 127.29.124.132:38673: session id = 953c332688d14831b3bc7cfd895e06e9-53249c71dab94ff4b179309088f4d1cd
I20240501 14:04:39.495671 1343 tablet_copy_source_session.cc:205] T 53249c71dab94ff4b179309088f4d1cd P 05f4ef3d126d43e9be32731e708ff76c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:04:39.498209 3354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53249c71dab94ff4b179309088f4d1cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:39.506605 3354 tablet_copy_client.cc:783] T 53249c71dab94ff4b179309088f4d1cd P 953c332688d14831b3bc7cfd895e06e9: tablet copy: Starting download of 0 data blocks...
I20240501 14:04:39.507033 3354 tablet_copy_client.cc:647] T 53249c71dab94ff4b179309088f4d1cd P 953c332688d14831b3bc7cfd895e06e9: tablet copy: Starting download of 1 WAL segments...
I20240501 14:04:39.509203 3354 tablet_copy_client.cc:523] T 53249c71dab94ff4b179309088f4d1cd P 953c332688d14831b3bc7cfd895e06e9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:04:39.518813 3354 tablet_bootstrap.cc:492] T 53249c71dab94ff4b179309088f4d1cd P 953c332688d14831b3bc7cfd895e06e9: Bootstrap starting.
I20240501 14:04:39.528563 3260 tablet_bootstrap.cc:492] T 0f837cac32ad4aaa970124639138c2db P 752bc4569da647679d5df51641961840: Bootstrap replayed 1/1 log segments. Stats: ops{read=100 overwritten=0 applied=100 ignored=0} inserts{seen=772 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:04:39.533756 3260 tablet_bootstrap.cc:492] T 0f837cac32ad4aaa970124639138c2db P 752bc4569da647679d5df51641961840: Bootstrap complete.
I20240501 14:04:39.535072 3260 ts_tablet_manager.cc:1367] T 0f837cac32ad4aaa970124639138c2db P 752bc4569da647679d5df51641961840: Time spent bootstrapping tablet: real 1.088s user 0.145s sys 0.020s
I20240501 14:04:39.535948 3260 raft_consensus.cc:348] T 0f837cac32ad4aaa970124639138c2db P 752bc4569da647679d5df51641961840 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: true } }
I20240501 14:04:39.540004 3260 raft_consensus.cc:729] T 0f837cac32ad4aaa970124639138c2db P 752bc4569da647679d5df51641961840 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 752bc4569da647679d5df51641961840, State: Initialized, Role: LEARNER
I20240501 14:04:39.540335 3260 consensus_queue.cc:260] T 0f837cac32ad4aaa970124639138c2db P 752bc4569da647679d5df51641961840 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 100, Last appended: 1.100, Last appended by leader: 100, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 89 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: true } }
I20240501 14:04:39.540998 3260 ts_tablet_manager.cc:1398] T 0f837cac32ad4aaa970124639138c2db P 752bc4569da647679d5df51641961840: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240501 14:04:39.546419 1213 tablet_copy_service.cc:342] P 5d45680f41c84ac39bd97c6f00ac2b06: Request end of tablet copy session 752bc4569da647679d5df51641961840-0f837cac32ad4aaa970124639138c2db received from {username='slave'} at 127.29.124.135:47221
I20240501 14:04:39.546553 1213 tablet_copy_service.cc:433] P 5d45680f41c84ac39bd97c6f00ac2b06: ending tablet copy session 752bc4569da647679d5df51641961840-0f837cac32ad4aaa970124639138c2db on tablet 0f837cac32ad4aaa970124639138c2db with peer 752bc4569da647679d5df51641961840
I20240501 14:04:39.550375 1343 tablet_copy_service.cc:140] P 05f4ef3d126d43e9be32731e708ff76c: Received BeginTabletCopySession request for tablet 4c22d4541da544f5b812c6e5e8c2ea40 from peer 32064197afde4b79bf388b080c365d88 ({username='slave'} at 127.29.124.134:36175)
I20240501 14:04:39.550547 1343 tablet_copy_service.cc:161] P 05f4ef3d126d43e9be32731e708ff76c: Beginning new tablet copy session on tablet 4c22d4541da544f5b812c6e5e8c2ea40 from peer 32064197afde4b79bf388b080c365d88 at {username='slave'} at 127.29.124.134:36175: session id = 32064197afde4b79bf388b080c365d88-4c22d4541da544f5b812c6e5e8c2ea40
I20240501 14:04:39.553735 1343 tablet_copy_source_session.cc:205] T 4c22d4541da544f5b812c6e5e8c2ea40 P 05f4ef3d126d43e9be32731e708ff76c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:04:39.554944 3249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c22d4541da544f5b812c6e5e8c2ea40. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:39.561363 3249 tablet_copy_client.cc:783] T 4c22d4541da544f5b812c6e5e8c2ea40 P 32064197afde4b79bf388b080c365d88: tablet copy: Starting download of 0 data blocks...
I20240501 14:04:39.561614 3249 tablet_copy_client.cc:647] T 4c22d4541da544f5b812c6e5e8c2ea40 P 32064197afde4b79bf388b080c365d88: tablet copy: Starting download of 1 WAL segments...
I20240501 14:04:39.563560 3249 tablet_copy_client.cc:523] T 4c22d4541da544f5b812c6e5e8c2ea40 P 32064197afde4b79bf388b080c365d88: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:04:39.566831 3249 tablet_bootstrap.cc:492] T 4c22d4541da544f5b812c6e5e8c2ea40 P 32064197afde4b79bf388b080c365d88: Bootstrap starting.
I20240501 14:04:39.618310 2885 raft_consensus.cc:1203] T 0f837cac32ad4aaa970124639138c2db P 752bc4569da647679d5df51641961840 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.88->[1.89-1.116] Dedup: 1.100->[1.101-1.116]
I20240501 14:04:39.685345 3170 raft_consensus.cc:1050] T 0f837cac32ad4aaa970124639138c2db P 5d45680f41c84ac39bd97c6f00ac2b06: attempting to promote NON_VOTER 752bc4569da647679d5df51641961840 to VOTER
I20240501 14:04:39.686041 3170 consensus_queue.cc:237] T 0f837cac32ad4aaa970124639138c2db P 5d45680f41c84ac39bd97c6f00ac2b06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 116, Committed index: 116, Last appended: 1.116, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 117 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false } }
I20240501 14:04:39.687882 2884 raft_consensus.cc:1261] T 0f837cac32ad4aaa970124639138c2db P 752bc4569da647679d5df51641961840 [term 1 LEARNER]: Refusing update from remote peer 5d45680f41c84ac39bd97c6f00ac2b06: Log matching property violated. Preceding OpId in replica: term: 1 index: 116. Preceding OpId from leader: term: 1 index: 117. (index mismatch)
I20240501 14:04:39.689514 2392 consensus_queue.cc:1035] T 0f837cac32ad4aaa970124639138c2db P 5d45680f41c84ac39bd97c6f00ac2b06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 117, Last known committed idx: 116, Time since last communication: 0.000s
I20240501 14:04:39.696544 1449 raft_consensus.cc:1261] T 0f837cac32ad4aaa970124639138c2db P eb004c2ac9d244f89b61448584d4115b [term 1 FOLLOWER]: Refusing update from remote peer 5d45680f41c84ac39bd97c6f00ac2b06: Log matching property violated. Preceding OpId in replica: term: 1 index: 116. Preceding OpId from leader: term: 1 index: 117. (index mismatch)
I20240501 14:04:39.701514 3153 raft_consensus.cc:2932] T 0f837cac32ad4aaa970124639138c2db P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 LEADER]: Committing config change with OpId 1.117: config changed from index 89 to 117, 752bc4569da647679d5df51641961840 (127.29.124.135) changed from NON_VOTER to VOTER. New config: { opid_index: 117 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false } } }
I20240501 14:04:39.706125 2885 raft_consensus.cc:2932] T 0f837cac32ad4aaa970124639138c2db P 752bc4569da647679d5df51641961840 [term 1 FOLLOWER]: Committing config change with OpId 1.117: config changed from index 89 to 117, 752bc4569da647679d5df51641961840 (127.29.124.135) changed from NON_VOTER to VOTER. New config: { opid_index: 117 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false } } }
I20240501 14:04:39.706938 1045 catalog_manager.cc:5484] T 0f837cac32ad4aaa970124639138c2db P 5d45680f41c84ac39bd97c6f00ac2b06 reported cstate change: config changed from index 89 to 117, 752bc4569da647679d5df51641961840 (127.29.124.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" committed_config { opid_index: 117 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:04:39.730391 3153 consensus_queue.cc:1035] T 0f837cac32ad4aaa970124639138c2db P 5d45680f41c84ac39bd97c6f00ac2b06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 117, Last known committed idx: 116, Time since last communication: 0.000s
I20240501 14:04:39.753063 1449 raft_consensus.cc:2932] T 0f837cac32ad4aaa970124639138c2db P eb004c2ac9d244f89b61448584d4115b [term 1 FOLLOWER]: Committing config change with OpId 1.117: config changed from index 89 to 117, 752bc4569da647679d5df51641961840 (127.29.124.135) changed from NON_VOTER to VOTER. New config: { opid_index: 117 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false } } }
I20240501 14:04:39.952069 3354 tablet_bootstrap.cc:492] T 53249c71dab94ff4b179309088f4d1cd P 953c332688d14831b3bc7cfd895e06e9: Bootstrap replayed 1/1 log segments. Stats: ops{read=90 overwritten=0 applied=90 ignored=0} inserts{seen=762 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:04:39.952666 3354 tablet_bootstrap.cc:492] T 53249c71dab94ff4b179309088f4d1cd P 953c332688d14831b3bc7cfd895e06e9: Bootstrap complete.
I20240501 14:04:39.953035 3354 ts_tablet_manager.cc:1367] T 53249c71dab94ff4b179309088f4d1cd P 953c332688d14831b3bc7cfd895e06e9: Time spent bootstrapping tablet: real 0.434s user 0.151s sys 0.001s
I20240501 14:04:39.954000 3354 raft_consensus.cc:348] T 53249c71dab94ff4b179309088f4d1cd P 953c332688d14831b3bc7cfd895e06e9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: NON_VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: true } }
I20240501 14:04:39.954435 3354 raft_consensus.cc:729] T 53249c71dab94ff4b179309088f4d1cd P 953c332688d14831b3bc7cfd895e06e9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 953c332688d14831b3bc7cfd895e06e9, State: Initialized, Role: LEARNER
I20240501 14:04:39.954730 3354 consensus_queue.cc:260] T 53249c71dab94ff4b179309088f4d1cd P 953c332688d14831b3bc7cfd895e06e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 90, Last appended: 1.90, Last appended by leader: 90, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 80 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: NON_VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: true } }
I20240501 14:04:39.983738 3354 ts_tablet_manager.cc:1398] T 53249c71dab94ff4b179309088f4d1cd P 953c332688d14831b3bc7cfd895e06e9: Time spent starting tablet: real 0.030s user 0.014s sys 0.000s
I20240501 14:04:39.987164 3238 tablet_bootstrap.cc:492] T 6a5325b7aa824831a5fa056fb4439110 P 953c332688d14831b3bc7cfd895e06e9: Bootstrap replayed 1/1 log segments. Stats: ops{read=85 overwritten=0 applied=85 ignored=0} inserts{seen=757 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:04:39.987725 3238 tablet_bootstrap.cc:492] T 6a5325b7aa824831a5fa056fb4439110 P 953c332688d14831b3bc7cfd895e06e9: Bootstrap complete.
I20240501 14:04:39.994324 3238 ts_tablet_manager.cc:1367] T 6a5325b7aa824831a5fa056fb4439110 P 953c332688d14831b3bc7cfd895e06e9: Time spent bootstrapping tablet: real 1.354s user 0.147s sys 0.002s
I20240501 14:04:39.996547 3238 raft_consensus.cc:348] T 6a5325b7aa824831a5fa056fb4439110 P 953c332688d14831b3bc7cfd895e06e9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: NON_VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: true } }
I20240501 14:04:39.996870 3238 raft_consensus.cc:729] T 6a5325b7aa824831a5fa056fb4439110 P 953c332688d14831b3bc7cfd895e06e9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 953c332688d14831b3bc7cfd895e06e9, State: Initialized, Role: LEARNER
I20240501 14:04:39.997156 3238 consensus_queue.cc:260] T 6a5325b7aa824831a5fa056fb4439110 P 953c332688d14831b3bc7cfd895e06e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 85, Last appended: 1.85, Last appended by leader: 85, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 79 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: NON_VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: true } }
I20240501 14:04:39.997768 3238 ts_tablet_manager.cc:1398] T 6a5325b7aa824831a5fa056fb4439110 P 953c332688d14831b3bc7cfd895e06e9: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:04:39.997756 1343 tablet_copy_service.cc:342] P 05f4ef3d126d43e9be32731e708ff76c: Request end of tablet copy session 953c332688d14831b3bc7cfd895e06e9-53249c71dab94ff4b179309088f4d1cd received from {username='slave'} at 127.29.124.132:38673
I20240501 14:04:39.997879 1343 tablet_copy_service.cc:433] P 05f4ef3d126d43e9be32731e708ff76c: ending tablet copy session 953c332688d14831b3bc7cfd895e06e9-53249c71dab94ff4b179309088f4d1cd on tablet 53249c71dab94ff4b179309088f4d1cd with peer 953c332688d14831b3bc7cfd895e06e9
I20240501 14:04:39.998497 1473 tablet_copy_service.cc:342] P eb004c2ac9d244f89b61448584d4115b: Request end of tablet copy session 953c332688d14831b3bc7cfd895e06e9-6a5325b7aa824831a5fa056fb4439110 received from {username='slave'} at 127.29.124.132:54979
I20240501 14:04:39.998637 1473 tablet_copy_service.cc:433] P eb004c2ac9d244f89b61448584d4115b: ending tablet copy session 953c332688d14831b3bc7cfd895e06e9-6a5325b7aa824831a5fa056fb4439110 on tablet 6a5325b7aa824831a5fa056fb4439110 with peer 953c332688d14831b3bc7cfd895e06e9
I20240501 14:04:40.046452 3269 tablet_bootstrap.cc:492] T 45d428dc499a4270b56a579f5b3cad83 P 4e6923f9753a4450ac4343c34c427d1e: Bootstrap replayed 1/1 log segments. Stats: ops{read=75 overwritten=0 applied=75 ignored=0} inserts{seen=747 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:04:40.046882 3269 tablet_bootstrap.cc:492] T 45d428dc499a4270b56a579f5b3cad83 P 4e6923f9753a4450ac4343c34c427d1e: Bootstrap complete.
I20240501 14:04:40.047150 3269 ts_tablet_manager.cc:1367] T 45d428dc499a4270b56a579f5b3cad83 P 4e6923f9753a4450ac4343c34c427d1e: Time spent bootstrapping tablet: real 1.571s user 0.148s sys 0.031s
I20240501 14:04:40.047895 3269 raft_consensus.cc:348] T 45d428dc499a4270b56a579f5b3cad83 P 4e6923f9753a4450ac4343c34c427d1e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: true } }
I20240501 14:04:40.048144 3269 raft_consensus.cc:729] T 45d428dc499a4270b56a579f5b3cad83 P 4e6923f9753a4450ac4343c34c427d1e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4e6923f9753a4450ac4343c34c427d1e, State: Initialized, Role: LEARNER
I20240501 14:04:40.048362 3269 consensus_queue.cc:260] T 45d428dc499a4270b56a579f5b3cad83 P 4e6923f9753a4450ac4343c34c427d1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 75, Last appended: 1.75, Last appended by leader: 75, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 72 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: true } }
I20240501 14:04:40.072105 3319 tablet_bootstrap.cc:492] T 70afcb72943b49a7bea962791b38a91d P 32064197afde4b79bf388b080c365d88: Bootstrap replayed 1/1 log segments. Stats: ops{read=86 overwritten=0 applied=86 ignored=0} inserts{seen=717 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:04:40.072644 3319 tablet_bootstrap.cc:492] T 70afcb72943b49a7bea962791b38a91d P 32064197afde4b79bf388b080c365d88: Bootstrap complete.
I20240501 14:04:40.073025 3319 ts_tablet_manager.cc:1367] T 70afcb72943b49a7bea962791b38a91d P 32064197afde4b79bf388b080c365d88: Time spent bootstrapping tablet: real 1.187s user 0.143s sys 0.000s
I20240501 14:04:40.073866 3319 raft_consensus.cc:348] T 70afcb72943b49a7bea962791b38a91d P 32064197afde4b79bf388b080c365d88 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } } peers { permanent_uuid: "32064197afde4b79bf388b080c365d88" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 37211 } attrs { promote: true } }
I20240501 14:04:40.074642 3319 raft_consensus.cc:729] T 70afcb72943b49a7bea962791b38a91d P 32064197afde4b79bf388b080c365d88 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 32064197afde4b79bf388b080c365d88, State: Initialized, Role: LEARNER
I20240501 14:04:40.074904 3319 consensus_queue.cc:260] T 70afcb72943b49a7bea962791b38a91d P 32064197afde4b79bf388b080c365d88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 86, Last appended: 1.86, Last appended by leader: 86, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 84 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } } peers { permanent_uuid: "32064197afde4b79bf388b080c365d88" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 37211 } attrs { promote: true } }
I20240501 14:04:40.078461 3269 ts_tablet_manager.cc:1398] T 45d428dc499a4270b56a579f5b3cad83 P 4e6923f9753a4450ac4343c34c427d1e: Time spent starting tablet: real 0.031s user 0.003s sys 0.000s
I20240501 14:04:40.080665 1473 tablet_copy_service.cc:342] P eb004c2ac9d244f89b61448584d4115b: Request end of tablet copy session 4e6923f9753a4450ac4343c34c427d1e-45d428dc499a4270b56a579f5b3cad83 received from {username='slave'} at 127.29.124.133:45461
I20240501 14:04:40.080806 1473 tablet_copy_service.cc:433] P eb004c2ac9d244f89b61448584d4115b: ending tablet copy session 4e6923f9753a4450ac4343c34c427d1e-45d428dc499a4270b56a579f5b3cad83 on tablet 45d428dc499a4270b56a579f5b3cad83 with peer 4e6923f9753a4450ac4343c34c427d1e
I20240501 14:04:40.091562 2488 raft_consensus.cc:1203] T 53249c71dab94ff4b179309088f4d1cd P 953c332688d14831b3bc7cfd895e06e9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.79->[1.80-1.95] Dedup: 1.90->[1.91-1.95]
I20240501 14:04:40.120488 2489 raft_consensus.cc:1203] T 6a5325b7aa824831a5fa056fb4439110 P 953c332688d14831b3bc7cfd895e06e9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.78->[1.79-1.97] Dedup: 1.85->[1.86-1.97]
I20240501 14:04:40.169687 3319 ts_tablet_manager.cc:1398] T 70afcb72943b49a7bea962791b38a91d P 32064197afde4b79bf388b080c365d88: Time spent starting tablet: real 0.096s user 0.017s sys 0.013s
I20240501 14:04:40.175017 3249 tablet_bootstrap.cc:492] T 4c22d4541da544f5b812c6e5e8c2ea40 P 32064197afde4b79bf388b080c365d88: Bootstrap replayed 1/1 log segments. Stats: ops{read=103 overwritten=0 applied=102 ignored=0} inserts{seen=787 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 14:04:40.176100 3249 tablet_bootstrap.cc:492] T 4c22d4541da544f5b812c6e5e8c2ea40 P 32064197afde4b79bf388b080c365d88: Bootstrap complete.
I20240501 14:04:40.176683 3214 tablet_bootstrap.cc:492] T db2b3f73d20c4790808bbe0f1d5e489c P 752bc4569da647679d5df51641961840: Bootstrap replayed 1/1 log segments. Stats: ops{read=89 overwritten=0 applied=88 ignored=0} inserts{seen=773 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 14:04:40.177155 3214 tablet_bootstrap.cc:492] T db2b3f73d20c4790808bbe0f1d5e489c P 752bc4569da647679d5df51641961840: Bootstrap complete.
I20240501 14:04:40.178755 3249 ts_tablet_manager.cc:1367] T 4c22d4541da544f5b812c6e5e8c2ea40 P 32064197afde4b79bf388b080c365d88: Time spent bootstrapping tablet: real 0.612s user 0.142s sys 0.019s
I20240501 14:04:40.179953 3249 raft_consensus.cc:348] T 4c22d4541da544f5b812c6e5e8c2ea40 P 32064197afde4b79bf388b080c365d88 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } } peers { permanent_uuid: "32064197afde4b79bf388b080c365d88" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 37211 } attrs { promote: true } }
I20240501 14:04:40.196988 3214 ts_tablet_manager.cc:1367] T db2b3f73d20c4790808bbe0f1d5e489c P 752bc4569da647679d5df51641961840: Time spent bootstrapping tablet: real 1.071s user 0.140s sys 0.011s
I20240501 14:04:40.199805 3214 raft_consensus.cc:348] T db2b3f73d20c4790808bbe0f1d5e489c P 752bc4569da647679d5df51641961840 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: true } }
I20240501 14:04:40.211453 3214 raft_consensus.cc:729] T db2b3f73d20c4790808bbe0f1d5e489c P 752bc4569da647679d5df51641961840 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 752bc4569da647679d5df51641961840, State: Initialized, Role: LEARNER
I20240501 14:04:40.211890 3214 consensus_queue.cc:260] T db2b3f73d20c4790808bbe0f1d5e489c P 752bc4569da647679d5df51641961840 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 88, Last appended: 1.89, Last appended by leader: 89, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 87 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: true } }
I20240501 14:04:40.214177 3249 raft_consensus.cc:729] T 4c22d4541da544f5b812c6e5e8c2ea40 P 32064197afde4b79bf388b080c365d88 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 32064197afde4b79bf388b080c365d88, State: Initialized, Role: LEARNER
I20240501 14:04:40.215684 3249 consensus_queue.cc:260] T 4c22d4541da544f5b812c6e5e8c2ea40 P 32064197afde4b79bf388b080c365d88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 102, Last appended: 1.103, Last appended by leader: 103, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 96 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } } peers { permanent_uuid: "32064197afde4b79bf388b080c365d88" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 37211 } attrs { promote: true } }
I20240501 14:04:40.216300 3214 ts_tablet_manager.cc:1398] T db2b3f73d20c4790808bbe0f1d5e489c P 752bc4569da647679d5df51641961840: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20240501 14:04:40.216483 3249 ts_tablet_manager.cc:1398] T 4c22d4541da544f5b812c6e5e8c2ea40 P 32064197afde4b79bf388b080c365d88: Time spent starting tablet: real 0.037s user 0.002s sys 0.001s
I20240501 14:04:40.222038 1342 tablet_copy_service.cc:342] P 05f4ef3d126d43e9be32731e708ff76c: Request end of tablet copy session 32064197afde4b79bf388b080c365d88-4c22d4541da544f5b812c6e5e8c2ea40 received from {username='slave'} at 127.29.124.134:36175
I20240501 14:04:40.222214 1342 tablet_copy_service.cc:433] P 05f4ef3d126d43e9be32731e708ff76c: ending tablet copy session 32064197afde4b79bf388b080c365d88-4c22d4541da544f5b812c6e5e8c2ea40 on tablet 4c22d4541da544f5b812c6e5e8c2ea40 with peer 32064197afde4b79bf388b080c365d88
I20240501 14:04:40.222654 3277 raft_consensus.cc:1050] T 53249c71dab94ff4b179309088f4d1cd P 05f4ef3d126d43e9be32731e708ff76c: attempting to promote NON_VOTER 953c332688d14831b3bc7cfd895e06e9 to VOTER
I20240501 14:04:40.222718 1343 tablet_copy_service.cc:342] P 05f4ef3d126d43e9be32731e708ff76c: Request end of tablet copy session 752bc4569da647679d5df51641961840-db2b3f73d20c4790808bbe0f1d5e489c received from {username='slave'} at 127.29.124.135:55917
I20240501 14:04:40.222839 1343 tablet_copy_service.cc:433] P 05f4ef3d126d43e9be32731e708ff76c: ending tablet copy session 752bc4569da647679d5df51641961840-db2b3f73d20c4790808bbe0f1d5e489c on tablet db2b3f73d20c4790808bbe0f1d5e489c with peer 752bc4569da647679d5df51641961840
I20240501 14:04:40.223412 3277 consensus_queue.cc:237] T 53249c71dab94ff4b179309088f4d1cd P 05f4ef3d126d43e9be32731e708ff76c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 100, Committed index: 100, Last appended: 1.100, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 101 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } }
I20240501 14:04:40.225489 2488 raft_consensus.cc:1261] T 53249c71dab94ff4b179309088f4d1cd P 953c332688d14831b3bc7cfd895e06e9 [term 1 LEARNER]: Refusing update from remote peer 05f4ef3d126d43e9be32731e708ff76c: Log matching property violated. Preceding OpId in replica: term: 1 index: 100. Preceding OpId from leader: term: 1 index: 101. (index mismatch)
I20240501 14:04:40.225636 1449 raft_consensus.cc:1261] T 53249c71dab94ff4b179309088f4d1cd P eb004c2ac9d244f89b61448584d4115b [term 1 FOLLOWER]: Refusing update from remote peer 05f4ef3d126d43e9be32731e708ff76c: Log matching property violated. Preceding OpId in replica: term: 1 index: 100. Preceding OpId from leader: term: 1 index: 101. (index mismatch)
I20240501 14:04:40.226236 3046 consensus_queue.cc:1035] T 53249c71dab94ff4b179309088f4d1cd P 05f4ef3d126d43e9be32731e708ff76c [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 101, Last known committed idx: 100, Time since last communication: 0.000s
I20240501 14:04:40.226881 3046 consensus_queue.cc:1035] T 53249c71dab94ff4b179309088f4d1cd P 05f4ef3d126d43e9be32731e708ff76c [LEADER]: Connected to new peer: Peer: permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 101, Last known committed idx: 100, Time since last communication: 0.000s
I20240501 14:04:40.231515 2287 raft_consensus.cc:2932] T 53249c71dab94ff4b179309088f4d1cd P 05f4ef3d126d43e9be32731e708ff76c [term 1 LEADER]: Committing config change with OpId 1.101: config changed from index 80 to 101, 953c332688d14831b3bc7cfd895e06e9 (127.29.124.132) changed from NON_VOTER to VOTER. New config: { opid_index: 101 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } } }
I20240501 14:04:40.236013 1449 raft_consensus.cc:2932] T 53249c71dab94ff4b179309088f4d1cd P eb004c2ac9d244f89b61448584d4115b [term 1 FOLLOWER]: Committing config change with OpId 1.101: config changed from index 80 to 101, 953c332688d14831b3bc7cfd895e06e9 (127.29.124.132) changed from NON_VOTER to VOTER. New config: { opid_index: 101 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } } }
I20240501 14:04:40.236923 1043 catalog_manager.cc:5484] T 53249c71dab94ff4b179309088f4d1cd P 05f4ef3d126d43e9be32731e708ff76c reported cstate change: config changed from index 80 to 101, 953c332688d14831b3bc7cfd895e06e9 (127.29.124.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "05f4ef3d126d43e9be32731e708ff76c" committed_config { opid_index: 101 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240501 14:04:40.242492 1129 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.124.131:40225, user_credentials={real_user=slave}} blocked reactor thread for 54058.1us
I20240501 14:04:40.247371 1213 tablet_copy_service.cc:342] P 5d45680f41c84ac39bd97c6f00ac2b06: Request end of tablet copy session 32064197afde4b79bf388b080c365d88-70afcb72943b49a7bea962791b38a91d received from {username='slave'} at 127.29.124.134:41391
I20240501 14:04:40.247509 1213 tablet_copy_service.cc:433] P 5d45680f41c84ac39bd97c6f00ac2b06: ending tablet copy session 32064197afde4b79bf388b080c365d88-70afcb72943b49a7bea962791b38a91d on tablet 70afcb72943b49a7bea962791b38a91d with peer 32064197afde4b79bf388b080c365d88
I20240501 14:04:40.248538 2752 raft_consensus.cc:1203] T 70afcb72943b49a7bea962791b38a91d P 32064197afde4b79bf388b080c365d88 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.83->[1.84-1.102] Dedup: 1.86->[1.87-1.102]
I20240501 14:04:40.256989 2488 raft_consensus.cc:2932] T 53249c71dab94ff4b179309088f4d1cd P 953c332688d14831b3bc7cfd895e06e9 [term 1 FOLLOWER]: Committing config change with OpId 1.101: config changed from index 80 to 101, 953c332688d14831b3bc7cfd895e06e9 (127.29.124.132) changed from NON_VOTER to VOTER. New config: { opid_index: 101 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } } }
I20240501 14:04:40.275884 2884 raft_consensus.cc:1203] T db2b3f73d20c4790808bbe0f1d5e489c P 752bc4569da647679d5df51641961840 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.86->[1.87-1.100] Dedup: 1.89->[1.90-1.100]
I20240501 14:04:40.279811 3171 raft_consensus.cc:1050] T 70afcb72943b49a7bea962791b38a91d P 5d45680f41c84ac39bd97c6f00ac2b06: attempting to promote NON_VOTER 32064197afde4b79bf388b080c365d88 to VOTER
I20240501 14:04:40.280565 3171 consensus_queue.cc:237] T 70afcb72943b49a7bea962791b38a91d P 5d45680f41c84ac39bd97c6f00ac2b06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 102, Committed index: 102, Last appended: 1.103, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } } peers { permanent_uuid: "32064197afde4b79bf388b080c365d88" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 37211 } attrs { promote: false } }
I20240501 14:04:40.282784 2752 raft_consensus.cc:1261] T 70afcb72943b49a7bea962791b38a91d P 32064197afde4b79bf388b080c365d88 [term 1 LEARNER]: Refusing update from remote peer 5d45680f41c84ac39bd97c6f00ac2b06: Log matching property violated. Preceding OpId in replica: term: 1 index: 103. Preceding OpId from leader: term: 1 index: 104. (index mismatch)
I20240501 14:04:40.283423 3153 consensus_queue.cc:1035] T 70afcb72943b49a7bea962791b38a91d P 5d45680f41c84ac39bd97c6f00ac2b06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32064197afde4b79bf388b080c365d88" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 37211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 104, Last known committed idx: 102, Time since last communication: 0.000s
I20240501 14:04:40.285930 3066 raft_consensus.cc:2932] T 70afcb72943b49a7bea962791b38a91d P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 LEADER]: Committing config change with OpId 1.104: config changed from index 84 to 104, 32064197afde4b79bf388b080c365d88 (127.29.124.134) changed from NON_VOTER to VOTER. New config: { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } } peers { permanent_uuid: "32064197afde4b79bf388b080c365d88" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 37211 } attrs { promote: false } } }
I20240501 14:04:40.291126 2621 raft_consensus.cc:1203] T 45d428dc499a4270b56a579f5b3cad83 P 4e6923f9753a4450ac4343c34c427d1e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.71->[1.72-1.91] Dedup: 1.75->[1.76-1.91]
I20240501 14:04:40.297319 1050 catalog_manager.cc:5484] T 70afcb72943b49a7bea962791b38a91d P 5d45680f41c84ac39bd97c6f00ac2b06 reported cstate change: config changed from index 84 to 104, 32064197afde4b79bf388b080c365d88 (127.29.124.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" committed_config { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "32064197afde4b79bf388b080c365d88" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 37211 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:04:40.302512 2752 raft_consensus.cc:2932] T 70afcb72943b49a7bea962791b38a91d P 32064197afde4b79bf388b080c365d88 [term 1 FOLLOWER]: Committing config change with OpId 1.104: config changed from index 84 to 104, 32064197afde4b79bf388b080c365d88 (127.29.124.134) changed from NON_VOTER to VOTER. New config: { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } } peers { permanent_uuid: "32064197afde4b79bf388b080c365d88" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 37211 } attrs { promote: false } } }
I20240501 14:04:40.320703 1450 raft_consensus.cc:1261] T 70afcb72943b49a7bea962791b38a91d P eb004c2ac9d244f89b61448584d4115b [term 1 FOLLOWER]: Refusing update from remote peer 5d45680f41c84ac39bd97c6f00ac2b06: Log matching property violated. Preceding OpId in replica: term: 1 index: 102. Preceding OpId from leader: term: 1 index: 104. (index mismatch)
I20240501 14:04:40.338240 3171 consensus_queue.cc:1035] T 70afcb72943b49a7bea962791b38a91d P 5d45680f41c84ac39bd97c6f00ac2b06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 104, Last known committed idx: 102, Time since last communication: 0.000s
I20240501 14:04:40.344977 1449 raft_consensus.cc:2932] T 70afcb72943b49a7bea962791b38a91d P eb004c2ac9d244f89b61448584d4115b [term 1 FOLLOWER]: Committing config change with OpId 1.104: config changed from index 84 to 104, 32064197afde4b79bf388b080c365d88 (127.29.124.134) changed from NON_VOTER to VOTER. New config: { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } } peers { permanent_uuid: "32064197afde4b79bf388b080c365d88" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 37211 } attrs { promote: false } } }
I20240501 14:04:40.362498 1192 consensus_queue.cc:237] T 70afcb72943b49a7bea962791b38a91d P 5d45680f41c84ac39bd97c6f00ac2b06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 106, Committed index: 106, Last appended: 1.107, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "32064197afde4b79bf388b080c365d88" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 37211 } attrs { promote: false } }
I20240501 14:04:40.367398 2751 raft_consensus.cc:1261] T 70afcb72943b49a7bea962791b38a91d P 32064197afde4b79bf388b080c365d88 [term 1 FOLLOWER]: Refusing update from remote peer 5d45680f41c84ac39bd97c6f00ac2b06: Log matching property violated. Preceding OpId in replica: term: 1 index: 107. Preceding OpId from leader: term: 1 index: 108. (index mismatch)
I20240501 14:04:40.368870 3066 consensus_queue.cc:1035] T 70afcb72943b49a7bea962791b38a91d P 5d45680f41c84ac39bd97c6f00ac2b06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32064197afde4b79bf388b080c365d88" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 37211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 108, Last known committed idx: 106, Time since last communication: 0.000s
I20240501 14:04:40.372336 3153 raft_consensus.cc:2932] T 70afcb72943b49a7bea962791b38a91d P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 LEADER]: Committing config change with OpId 1.108: config changed from index 104 to 108, VOTER eb004c2ac9d244f89b61448584d4115b (127.29.124.131) evicted. New config: { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "32064197afde4b79bf388b080c365d88" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 37211 } attrs { promote: false } } }
I20240501 14:04:40.381580 1050 catalog_manager.cc:5484] T 70afcb72943b49a7bea962791b38a91d P 5d45680f41c84ac39bd97c6f00ac2b06 reported cstate change: config changed from index 104 to 108, VOTER eb004c2ac9d244f89b61448584d4115b (127.29.124.131) evicted. New cstate: current_term: 1 leader_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" committed_config { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "32064197afde4b79bf388b080c365d88" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 37211 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:04:40.383571 2752 raft_consensus.cc:2932] T 70afcb72943b49a7bea962791b38a91d P 32064197afde4b79bf388b080c365d88 [term 1 FOLLOWER]: Committing config change with OpId 1.108: config changed from index 104 to 108, VOTER eb004c2ac9d244f89b61448584d4115b (127.29.124.131) evicted. New config: { opid_index: 108 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } } peers { permanent_uuid: "32064197afde4b79bf388b080c365d88" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 37211 } attrs { promote: false } } }
I20240501 14:04:40.389384 1036 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 70afcb72943b49a7bea962791b38a91d with cas_config_opid_index 104: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:04:40.391831 1432 tablet_service.cc:1514] Processing DeleteTablet for tablet 70afcb72943b49a7bea962791b38a91d with delete_type TABLET_DATA_TOMBSTONED (TS eb004c2ac9d244f89b61448584d4115b not found in new config with opid_index 108) from {username='slave'} at 127.0.0.1:38992
I20240501 14:04:40.420228 3412 tablet_replica.cc:330] T 70afcb72943b49a7bea962791b38a91d P eb004c2ac9d244f89b61448584d4115b: stopping tablet replica
I20240501 14:04:40.421015 3412 raft_consensus.cc:2229] T 70afcb72943b49a7bea962791b38a91d P eb004c2ac9d244f89b61448584d4115b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:04:40.422418 3412 pending_rounds.cc:62] T 70afcb72943b49a7bea962791b38a91d P eb004c2ac9d244f89b61448584d4115b: Trying to abort 1 pending ops.
I20240501 14:04:40.422583 3412 pending_rounds.cc:68] T 70afcb72943b49a7bea962791b38a91d P eb004c2ac9d244f89b61448584d4115b: Aborting op as it isn't in flight: id { term: 1 index: 107 } timestamp: 7022888060346253312 op_type: WRITE_OP write_request { tablet_id: "70afcb72943b49a7bea962791b38a91d" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\210z6(\362F\006&\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""d5a4055886b600aa" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7022888060252606464 authz_token { token_data: "\010\236\232\311\261\006\"3\n\005slave\022*\n 9c1d190b2b264f27af23180ab8f30d42\020\001\030\001 \001(\001" signature: "<redacted>""$\201Yf\257\023\314\334\356\211U,\244\337\250 \235\366\261n\032\243\343\201\373\202\312\177r\357\347{\270\236&?:\307$\2018\373\324B\274\221\036\027P\2621\224\236\3114\260\234\255\3148\277\nY\265" signing_key_seq_num: 0 } } request_id { client_id: "639433f7ae4744398c799eadc633e8ab" seq_no: 316 first_incomplete_seq_no: 316 attempt_no: 0 }
I20240501 14:04:40.425592 3412 raft_consensus.cc:2258] T 70afcb72943b49a7bea962791b38a91d P eb004c2ac9d244f89b61448584d4115b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:04:40.426990 2970 raft_consensus.cc:1050] T 6a5325b7aa824831a5fa056fb4439110 P eb004c2ac9d244f89b61448584d4115b: attempting to promote NON_VOTER 953c332688d14831b3bc7cfd895e06e9 to VOTER
I20240501 14:04:40.427440 3412 ts_tablet_manager.cc:1875] T 70afcb72943b49a7bea962791b38a91d P eb004c2ac9d244f89b61448584d4115b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:04:40.427687 2970 consensus_queue.cc:237] T 6a5325b7aa824831a5fa056fb4439110 P eb004c2ac9d244f89b61448584d4115b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 98, Committed index: 98, Last appended: 1.98, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } }
I20240501 14:04:40.429634 1322 raft_consensus.cc:1261] T 6a5325b7aa824831a5fa056fb4439110 P 05f4ef3d126d43e9be32731e708ff76c [term 1 FOLLOWER]: Refusing update from remote peer eb004c2ac9d244f89b61448584d4115b: Log matching property violated. Preceding OpId in replica: term: 1 index: 98. Preceding OpId from leader: term: 1 index: 99. (index mismatch)
I20240501 14:04:40.430073 2488 raft_consensus.cc:1261] T 6a5325b7aa824831a5fa056fb4439110 P 953c332688d14831b3bc7cfd895e06e9 [term 1 LEARNER]: Refusing update from remote peer eb004c2ac9d244f89b61448584d4115b: Log matching property violated. Preceding OpId in replica: term: 1 index: 98. Preceding OpId from leader: term: 1 index: 99. (index mismatch)
I20240501 14:04:40.430516 3201 consensus_queue.cc:1035] T 6a5325b7aa824831a5fa056fb4439110 P eb004c2ac9d244f89b61448584d4115b [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 99, Last known committed idx: 98, Time since last communication: 0.000s
I20240501 14:04:40.430920 3201 consensus_queue.cc:1035] T 6a5325b7aa824831a5fa056fb4439110 P eb004c2ac9d244f89b61448584d4115b [LEADER]: Connected to new peer: Peer: permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 99, Last known committed idx: 98, Time since last communication: 0.000s
I20240501 14:04:40.437155 2752 raft_consensus.cc:1203] T 4c22d4541da544f5b812c6e5e8c2ea40 P 32064197afde4b79bf388b080c365d88 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.95->[1.96-1.115] Dedup: 1.103->[1.104-1.115]
I20240501 14:04:40.439749 3261 raft_consensus.cc:2932] T 6a5325b7aa824831a5fa056fb4439110 P eb004c2ac9d244f89b61448584d4115b [term 1 LEADER]: Committing config change with OpId 1.99: config changed from index 79 to 99, 953c332688d14831b3bc7cfd895e06e9 (127.29.124.132) changed from NON_VOTER to VOTER. New config: { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } } }
I20240501 14:04:40.446424 2487 raft_consensus.cc:2932] T 6a5325b7aa824831a5fa056fb4439110 P 953c332688d14831b3bc7cfd895e06e9 [term 1 FOLLOWER]: Committing config change with OpId 1.99: config changed from index 79 to 99, 953c332688d14831b3bc7cfd895e06e9 (127.29.124.132) changed from NON_VOTER to VOTER. New config: { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } } }
I20240501 14:04:40.454336 1052 catalog_manager.cc:5484] T 6a5325b7aa824831a5fa056fb4439110 P 953c332688d14831b3bc7cfd895e06e9 reported cstate change: config changed from index 79 to 99, 953c332688d14831b3bc7cfd895e06e9 (127.29.124.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "eb004c2ac9d244f89b61448584d4115b" committed_config { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } } }
I20240501 14:04:40.456820 1323 raft_consensus.cc:2932] T 6a5325b7aa824831a5fa056fb4439110 P 05f4ef3d126d43e9be32731e708ff76c [term 1 FOLLOWER]: Committing config change with OpId 1.99: config changed from index 79 to 99, 953c332688d14831b3bc7cfd895e06e9 (127.29.124.132) changed from NON_VOTER to VOTER. New config: { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } } }
I20240501 14:04:40.459076 3201 raft_consensus.cc:1050] T 45d428dc499a4270b56a579f5b3cad83 P eb004c2ac9d244f89b61448584d4115b: attempting to promote NON_VOTER 4e6923f9753a4450ac4343c34c427d1e to VOTER
I20240501 14:04:40.460234 3201 consensus_queue.cc:237] T 45d428dc499a4270b56a579f5b3cad83 P eb004c2ac9d244f89b61448584d4115b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 91, Committed index: 91, Last appended: 1.92, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: false } }
I20240501 14:04:40.462096 2622 raft_consensus.cc:1261] T 45d428dc499a4270b56a579f5b3cad83 P 4e6923f9753a4450ac4343c34c427d1e [term 1 LEARNER]: Refusing update from remote peer eb004c2ac9d244f89b61448584d4115b: Log matching property violated. Preceding OpId in replica: term: 1 index: 92. Preceding OpId from leader: term: 1 index: 93. (index mismatch)
I20240501 14:04:40.463783 3201 consensus_queue.cc:1035] T 45d428dc499a4270b56a579f5b3cad83 P eb004c2ac9d244f89b61448584d4115b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 93, Last known committed idx: 91, Time since last communication: 0.000s
I20240501 14:04:40.473951 1323 raft_consensus.cc:1261] T 45d428dc499a4270b56a579f5b3cad83 P 05f4ef3d126d43e9be32731e708ff76c [term 1 FOLLOWER]: Refusing update from remote peer eb004c2ac9d244f89b61448584d4115b: Log matching property violated. Preceding OpId in replica: term: 1 index: 92. Preceding OpId from leader: term: 1 index: 93. (index mismatch)
I20240501 14:04:40.478623 2970 raft_consensus.cc:2932] T 45d428dc499a4270b56a579f5b3cad83 P eb004c2ac9d244f89b61448584d4115b [term 1 LEADER]: Committing config change with OpId 1.93: config changed from index 72 to 93, 4e6923f9753a4450ac4343c34c427d1e (127.29.124.133) changed from NON_VOTER to VOTER. New config: { opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: false } } }
I20240501 14:04:40.484586 2622 raft_consensus.cc:2932] T 45d428dc499a4270b56a579f5b3cad83 P 4e6923f9753a4450ac4343c34c427d1e [term 1 FOLLOWER]: Committing config change with OpId 1.93: config changed from index 72 to 93, 4e6923f9753a4450ac4343c34c427d1e (127.29.124.133) changed from NON_VOTER to VOTER. New config: { opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: false } } }
I20240501 14:04:40.500021 3261 consensus_queue.cc:1035] T 45d428dc499a4270b56a579f5b3cad83 P eb004c2ac9d244f89b61448584d4115b [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 93, Last known committed idx: 91, Time since last communication: 0.000s
I20240501 14:04:40.503137 1323 raft_consensus.cc:2932] T 45d428dc499a4270b56a579f5b3cad83 P 05f4ef3d126d43e9be32731e708ff76c [term 1 FOLLOWER]: Committing config change with OpId 1.93: config changed from index 72 to 93, 4e6923f9753a4450ac4343c34c427d1e (127.29.124.133) changed from NON_VOTER to VOTER. New config: { opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: false } } }
I20240501 14:04:40.520367 1052 catalog_manager.cc:5484] T 45d428dc499a4270b56a579f5b3cad83 P eb004c2ac9d244f89b61448584d4115b reported cstate change: config changed from index 72 to 93, 4e6923f9753a4450ac4343c34c427d1e (127.29.124.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "eb004c2ac9d244f89b61448584d4115b" committed_config { opid_index: 93 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:04:40.526274 3277 raft_consensus.cc:1050] T db2b3f73d20c4790808bbe0f1d5e489c P 05f4ef3d126d43e9be32731e708ff76c: attempting to promote NON_VOTER 752bc4569da647679d5df51641961840 to VOTER
I20240501 14:04:40.526955 3277 consensus_queue.cc:237] T db2b3f73d20c4790808bbe0f1d5e489c P 05f4ef3d126d43e9be32731e708ff76c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 102, Committed index: 102, Last appended: 1.102, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false } }
I20240501 14:04:40.529852 1192 raft_consensus.cc:1261] T db2b3f73d20c4790808bbe0f1d5e489c P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 FOLLOWER]: Refusing update from remote peer 05f4ef3d126d43e9be32731e708ff76c: Log matching property violated. Preceding OpId in replica: term: 1 index: 102. Preceding OpId from leader: term: 1 index: 103. (index mismatch)
I20240501 14:04:40.530441 3409 consensus_queue.cc:1035] T db2b3f73d20c4790808bbe0f1d5e489c P 05f4ef3d126d43e9be32731e708ff76c [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 103, Last known committed idx: 102, Time since last communication: 0.000s
I20240501 14:04:40.531039 2884 raft_consensus.cc:1261] T db2b3f73d20c4790808bbe0f1d5e489c P 752bc4569da647679d5df51641961840 [term 1 LEARNER]: Refusing update from remote peer 05f4ef3d126d43e9be32731e708ff76c: Log matching property violated. Preceding OpId in replica: term: 1 index: 102. Preceding OpId from leader: term: 1 index: 103. (index mismatch)
I20240501 14:04:40.531893 1450 consensus_queue.cc:237] T 45d428dc499a4270b56a579f5b3cad83 P eb004c2ac9d244f89b61448584d4115b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 93, Committed index: 93, Last appended: 1.94, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: false } }
I20240501 14:04:40.532507 3046 consensus_queue.cc:1035] T db2b3f73d20c4790808bbe0f1d5e489c P 05f4ef3d126d43e9be32731e708ff76c [LEADER]: Connected to new peer: Peer: permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 103, Last known committed idx: 102, Time since last communication: 0.000s
I20240501 14:04:40.533331 3046 raft_consensus.cc:2932] T db2b3f73d20c4790808bbe0f1d5e489c P 05f4ef3d126d43e9be32731e708ff76c [term 1 LEADER]: Committing config change with OpId 1.103: config changed from index 87 to 103, 752bc4569da647679d5df51641961840 (127.29.124.135) changed from NON_VOTER to VOTER. New config: { opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false } } }
I20240501 14:04:40.536661 2621 raft_consensus.cc:1261] T 45d428dc499a4270b56a579f5b3cad83 P 4e6923f9753a4450ac4343c34c427d1e [term 1 FOLLOWER]: Refusing update from remote peer eb004c2ac9d244f89b61448584d4115b: Log matching property violated. Preceding OpId in replica: term: 1 index: 93. Preceding OpId from leader: term: 1 index: 95. (index mismatch)
I20240501 14:04:40.537277 3417 consensus_queue.cc:1035] T 45d428dc499a4270b56a579f5b3cad83 P eb004c2ac9d244f89b61448584d4115b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 95, Last known committed idx: 93, Time since last communication: 0.000s
I20240501 14:04:40.537591 2287 raft_consensus.cc:1050] T 4c22d4541da544f5b812c6e5e8c2ea40 P 05f4ef3d126d43e9be32731e708ff76c: attempting to promote NON_VOTER 32064197afde4b79bf388b080c365d88 to VOTER
I20240501 14:04:40.538262 2287 consensus_queue.cc:237] T 4c22d4541da544f5b812c6e5e8c2ea40 P 05f4ef3d126d43e9be32731e708ff76c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 116, Committed index: 116, Last appended: 1.117, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 118 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } } peers { permanent_uuid: "32064197afde4b79bf388b080c365d88" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 37211 } attrs { promote: false } }
I20240501 14:04:40.542301 1052 catalog_manager.cc:5484] T db2b3f73d20c4790808bbe0f1d5e489c P 05f4ef3d126d43e9be32731e708ff76c reported cstate change: config changed from index 87 to 103, 752bc4569da647679d5df51641961840 (127.29.124.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "05f4ef3d126d43e9be32731e708ff76c" committed_config { opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:04:40.542629 1192 raft_consensus.cc:2932] T db2b3f73d20c4790808bbe0f1d5e489c P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 FOLLOWER]: Committing config change with OpId 1.103: config changed from index 87 to 103, 752bc4569da647679d5df51641961840 (127.29.124.135) changed from NON_VOTER to VOTER. New config: { opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false } } }
I20240501 14:04:40.543828 2752 raft_consensus.cc:1261] T 4c22d4541da544f5b812c6e5e8c2ea40 P 32064197afde4b79bf388b080c365d88 [term 1 LEARNER]: Refusing update from remote peer 05f4ef3d126d43e9be32731e708ff76c: Log matching property violated. Preceding OpId in replica: term: 1 index: 117. Preceding OpId from leader: term: 1 index: 118. (index mismatch)
I20240501 14:04:40.545902 3261 raft_consensus.cc:2932] T 45d428dc499a4270b56a579f5b3cad83 P eb004c2ac9d244f89b61448584d4115b [term 1 LEADER]: Committing config change with OpId 1.95: config changed from index 93 to 95, VOTER 05f4ef3d126d43e9be32731e708ff76c (127.29.124.130) evicted. New config: { opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: false } } }
I20240501 14:04:40.546449 3412 ts_tablet_manager.cc:1888] T 70afcb72943b49a7bea962791b38a91d P eb004c2ac9d244f89b61448584d4115b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.107
I20240501 14:04:40.547855 3412 log.cc:1196] T 70afcb72943b49a7bea962791b38a91d P eb004c2ac9d244f89b61448584d4115b: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/wal/wals/70afcb72943b49a7bea962791b38a91d
I20240501 14:04:40.548646 3409 consensus_queue.cc:1035] T 4c22d4541da544f5b812c6e5e8c2ea40 P 05f4ef3d126d43e9be32731e708ff76c [LEADER]: Connected to new peer: Peer: permanent_uuid: "32064197afde4b79bf388b080c365d88" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 37211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 118, Last known committed idx: 116, Time since last communication: 0.000s
I20240501 14:04:40.549114 2621 raft_consensus.cc:2932] T 45d428dc499a4270b56a579f5b3cad83 P 4e6923f9753a4450ac4343c34c427d1e [term 1 FOLLOWER]: Committing config change with OpId 1.95: config changed from index 93 to 95, VOTER 05f4ef3d126d43e9be32731e708ff76c (127.29.124.130) evicted. New config: { opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: false } } }
I20240501 14:04:40.550257 2883 raft_consensus.cc:2932] T db2b3f73d20c4790808bbe0f1d5e489c P 752bc4569da647679d5df51641961840 [term 1 FOLLOWER]: Committing config change with OpId 1.103: config changed from index 87 to 103, 752bc4569da647679d5df51641961840 (127.29.124.135) changed from NON_VOTER to VOTER. New config: { opid_index: 103 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false } } }
I20240501 14:04:40.553167 1039 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 45d428dc499a4270b56a579f5b3cad83 with cas_config_opid_index 93: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:04:40.554795 1450 raft_consensus.cc:1261] T 4c22d4541da544f5b812c6e5e8c2ea40 P eb004c2ac9d244f89b61448584d4115b [term 1 FOLLOWER]: Refusing update from remote peer 05f4ef3d126d43e9be32731e708ff76c: Log matching property violated. Preceding OpId in replica: term: 1 index: 116. Preceding OpId from leader: term: 1 index: 118. (index mismatch)
I20240501 14:04:40.557018 1045 catalog_manager.cc:5484] T 45d428dc499a4270b56a579f5b3cad83 P eb004c2ac9d244f89b61448584d4115b reported cstate change: config changed from index 93 to 95, VOTER 05f4ef3d126d43e9be32731e708ff76c (127.29.124.130) evicted. New cstate: current_term: 1 leader_uuid: "eb004c2ac9d244f89b61448584d4115b" committed_config { opid_index: 95 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:04:40.557847 1323 consensus_queue.cc:237] T db2b3f73d20c4790808bbe0f1d5e489c P 05f4ef3d126d43e9be32731e708ff76c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 103, Committed index: 103, Last appended: 1.103, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false } }
I20240501 14:04:40.558633 1039 catalog_manager.cc:4830] TS eb004c2ac9d244f89b61448584d4115b (127.29.124.131:40225): tablet 70afcb72943b49a7bea962791b38a91d (table rebalance_test_table_2 [id=9c1d190b2b264f27af23180ab8f30d42]) successfully deleted
I20240501 14:04:40.562716 1302 tablet_service.cc:1514] Processing DeleteTablet for tablet 45d428dc499a4270b56a579f5b3cad83 with delete_type TABLET_DATA_TOMBSTONED (TS 05f4ef3d126d43e9be32731e708ff76c not found in new config with opid_index 95) from {username='slave'} at 127.0.0.1:35916
I20240501 14:04:40.562546 3046 raft_consensus.cc:2932] T 4c22d4541da544f5b812c6e5e8c2ea40 P 05f4ef3d126d43e9be32731e708ff76c [term 1 LEADER]: Committing config change with OpId 1.118: config changed from index 96 to 118, 32064197afde4b79bf388b080c365d88 (127.29.124.134) changed from NON_VOTER to VOTER. New config: { opid_index: 118 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } } peers { permanent_uuid: "32064197afde4b79bf388b080c365d88" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 37211 } attrs { promote: false } } }
I20240501 14:04:40.568945 3277 consensus_queue.cc:1035] T 4c22d4541da544f5b812c6e5e8c2ea40 P 05f4ef3d126d43e9be32731e708ff76c [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 118, Last known committed idx: 116, Time since last communication: 0.000s
I20240501 14:04:40.568886 2752 raft_consensus.cc:2932] T 4c22d4541da544f5b812c6e5e8c2ea40 P 32064197afde4b79bf388b080c365d88 [term 1 FOLLOWER]: Committing config change with OpId 1.118: config changed from index 96 to 118, 32064197afde4b79bf388b080c365d88 (127.29.124.134) changed from NON_VOTER to VOTER. New config: { opid_index: 118 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } } peers { permanent_uuid: "32064197afde4b79bf388b080c365d88" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 37211 } attrs { promote: false } } }
I20240501 14:04:40.576130 1045 catalog_manager.cc:5484] T 4c22d4541da544f5b812c6e5e8c2ea40 P 32064197afde4b79bf388b080c365d88 reported cstate change: config changed from index 96 to 118, 32064197afde4b79bf388b080c365d88 (127.29.124.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "05f4ef3d126d43e9be32731e708ff76c" committed_config { opid_index: 118 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } } peers { permanent_uuid: "32064197afde4b79bf388b080c365d88" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 37211 } attrs { promote: false } } }
I20240501 14:04:40.582010 1322 consensus_queue.cc:237] T 4c22d4541da544f5b812c6e5e8c2ea40 P 05f4ef3d126d43e9be32731e708ff76c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 118, Committed index: 118, Last appended: 1.118, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 119 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "32064197afde4b79bf388b080c365d88" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 37211 } attrs { promote: false } }
I20240501 14:04:40.586441 2752 raft_consensus.cc:1261] T 4c22d4541da544f5b812c6e5e8c2ea40 P 32064197afde4b79bf388b080c365d88 [term 1 FOLLOWER]: Refusing update from remote peer 05f4ef3d126d43e9be32731e708ff76c: Log matching property violated. Preceding OpId in replica: term: 1 index: 118. Preceding OpId from leader: term: 1 index: 119. (index mismatch)
I20240501 14:04:40.587141 2885 raft_consensus.cc:1261] T db2b3f73d20c4790808bbe0f1d5e489c P 752bc4569da647679d5df51641961840 [term 1 FOLLOWER]: Refusing update from remote peer 05f4ef3d126d43e9be32731e708ff76c: Log matching property violated. Preceding OpId in replica: term: 1 index: 103. Preceding OpId from leader: term: 1 index: 105. (index mismatch)
I20240501 14:04:40.588152 2287 consensus_queue.cc:1035] T db2b3f73d20c4790808bbe0f1d5e489c P 05f4ef3d126d43e9be32731e708ff76c [LEADER]: Connected to new peer: Peer: permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 104, Last known committed idx: 103, Time since last communication: 0.000s
I20240501 14:04:40.588789 2287 consensus_queue.cc:1035] T 4c22d4541da544f5b812c6e5e8c2ea40 P 05f4ef3d126d43e9be32731e708ff76c [LEADER]: Connected to new peer: Peer: permanent_uuid: "32064197afde4b79bf388b080c365d88" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 37211 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 119, Last known committed idx: 118, Time since last communication: 0.000s
I20240501 14:04:40.594537 3409 raft_consensus.cc:2932] T db2b3f73d20c4790808bbe0f1d5e489c P 05f4ef3d126d43e9be32731e708ff76c [term 1 LEADER]: Committing config change with OpId 1.104: config changed from index 103 to 104, VOTER 5d45680f41c84ac39bd97c6f00ac2b06 (127.29.124.129) evicted. New config: { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false } } }
I20240501 14:04:40.600222 3277 raft_consensus.cc:2932] T 4c22d4541da544f5b812c6e5e8c2ea40 P 05f4ef3d126d43e9be32731e708ff76c [term 1 LEADER]: Committing config change with OpId 1.119: config changed from index 118 to 119, VOTER eb004c2ac9d244f89b61448584d4115b (127.29.124.131) evicted. New config: { opid_index: 119 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "32064197afde4b79bf388b080c365d88" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 37211 } attrs { promote: false } } }
I20240501 14:04:40.603899 1038 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4c22d4541da544f5b812c6e5e8c2ea40 with cas_config_opid_index 118: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:04:40.606165 2752 raft_consensus.cc:2932] T 4c22d4541da544f5b812c6e5e8c2ea40 P 32064197afde4b79bf388b080c365d88 [term 1 FOLLOWER]: Committing config change with OpId 1.119: config changed from index 118 to 119, VOTER eb004c2ac9d244f89b61448584d4115b (127.29.124.131) evicted. New config: { opid_index: 119 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "32064197afde4b79bf388b080c365d88" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 37211 } attrs { promote: false } } }
I20240501 14:04:40.607033 2885 raft_consensus.cc:2932] T db2b3f73d20c4790808bbe0f1d5e489c P 752bc4569da647679d5df51641961840 [term 1 FOLLOWER]: Committing config change with OpId 1.104: config changed from index 103 to 104, VOTER 5d45680f41c84ac39bd97c6f00ac2b06 (127.29.124.129) evicted. New config: { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false } } }
I20240501 14:04:40.614049 1050 catalog_manager.cc:5484] T 4c22d4541da544f5b812c6e5e8c2ea40 P 32064197afde4b79bf388b080c365d88 reported cstate change: config changed from index 118 to 119, VOTER eb004c2ac9d244f89b61448584d4115b (127.29.124.131) evicted. New cstate: current_term: 1 leader_uuid: "05f4ef3d126d43e9be32731e708ff76c" committed_config { opid_index: 119 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "32064197afde4b79bf388b080c365d88" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 37211 } attrs { promote: false } } }
I20240501 14:04:40.619700 1432 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c22d4541da544f5b812c6e5e8c2ea40 with delete_type TABLET_DATA_TOMBSTONED (TS eb004c2ac9d244f89b61448584d4115b not found in new config with opid_index 119) from {username='slave'} at 127.0.0.1:38992
I20240501 14:04:40.619227 1449 raft_consensus.cc:2932] T 4c22d4541da544f5b812c6e5e8c2ea40 P eb004c2ac9d244f89b61448584d4115b [term 1 FOLLOWER]: Committing config change with OpId 1.118: config changed from index 96 to 118, 32064197afde4b79bf388b080c365d88 (127.29.124.134) changed from NON_VOTER to VOTER. New config: { opid_index: 118 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } } peers { permanent_uuid: "32064197afde4b79bf388b080c365d88" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 37211 } attrs { promote: false } } }
I20240501 14:04:40.640053 1045 catalog_manager.cc:5484] T db2b3f73d20c4790808bbe0f1d5e489c P 752bc4569da647679d5df51641961840 reported cstate change: config changed from index 103 to 104, VOTER 5d45680f41c84ac39bd97c6f00ac2b06 (127.29.124.129) evicted. New cstate: current_term: 1 leader_uuid: "05f4ef3d126d43e9be32731e708ff76c" committed_config { opid_index: 104 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false } } }
I20240501 14:04:40.644757 1172 tablet_service.cc:1514] Processing DeleteTablet for tablet db2b3f73d20c4790808bbe0f1d5e489c with delete_type TABLET_DATA_TOMBSTONED (TS 5d45680f41c84ac39bd97c6f00ac2b06 not found in new config with opid_index 104) from {username='slave'} at 127.0.0.1:51144
I20240501 14:04:40.677675 3412 tablet_replica.cc:330] T 4c22d4541da544f5b812c6e5e8c2ea40 P eb004c2ac9d244f89b61448584d4115b: stopping tablet replica
I20240501 14:04:40.677877 3412 raft_consensus.cc:2229] T 4c22d4541da544f5b812c6e5e8c2ea40 P eb004c2ac9d244f89b61448584d4115b [term 1 FOLLOWER]: Raft consensus shutting down.
W20240501 14:04:40.677892 1258 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.124.135:42923, user_credentials={real_user=slave}} blocked reactor thread for 62712.4us
I20240501 14:04:40.678519 3412 raft_consensus.cc:2258] T 4c22d4541da544f5b812c6e5e8c2ea40 P eb004c2ac9d244f89b61448584d4115b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:04:40.678797 1038 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet db2b3f73d20c4790808bbe0f1d5e489c with cas_config_opid_index 103: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:04:40.680001 3412 ts_tablet_manager.cc:1875] T 4c22d4541da544f5b812c6e5e8c2ea40 P eb004c2ac9d244f89b61448584d4115b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:04:40.687130 3412 ts_tablet_manager.cc:1888] T 4c22d4541da544f5b812c6e5e8c2ea40 P eb004c2ac9d244f89b61448584d4115b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.118
I20240501 14:04:40.687929 3412 log.cc:1196] T 4c22d4541da544f5b812c6e5e8c2ea40 P eb004c2ac9d244f89b61448584d4115b: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/wal/wals/4c22d4541da544f5b812c6e5e8c2ea40
I20240501 14:04:40.689093 1039 catalog_manager.cc:4830] TS eb004c2ac9d244f89b61448584d4115b (127.29.124.131:40225): tablet 4c22d4541da544f5b812c6e5e8c2ea40 (table rebalance_test_table_0 [id=817d8db2cf8f4cffb29e21f78e86e2d5]) successfully deleted
I20240501 14:04:40.695495 3428 tablet_replica.cc:330] T db2b3f73d20c4790808bbe0f1d5e489c P 5d45680f41c84ac39bd97c6f00ac2b06: stopping tablet replica
I20240501 14:04:40.695715 3428 raft_consensus.cc:2229] T db2b3f73d20c4790808bbe0f1d5e489c P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:04:40.696265 3428 raft_consensus.cc:2258] T db2b3f73d20c4790808bbe0f1d5e489c P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:04:40.696785 3424 tablet_replica.cc:330] T 45d428dc499a4270b56a579f5b3cad83 P 05f4ef3d126d43e9be32731e708ff76c: stopping tablet replica
I20240501 14:04:40.696969 3424 raft_consensus.cc:2229] T 45d428dc499a4270b56a579f5b3cad83 P 05f4ef3d126d43e9be32731e708ff76c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:04:40.697588 3424 raft_consensus.cc:2258] T 45d428dc499a4270b56a579f5b3cad83 P 05f4ef3d126d43e9be32731e708ff76c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:04:40.699360 3424 ts_tablet_manager.cc:1875] T 45d428dc499a4270b56a579f5b3cad83 P 05f4ef3d126d43e9be32731e708ff76c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:04:40.714078 3424 ts_tablet_manager.cc:1888] T 45d428dc499a4270b56a579f5b3cad83 P 05f4ef3d126d43e9be32731e708ff76c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.93
I20240501 14:04:40.714207 3424 log.cc:1196] T 45d428dc499a4270b56a579f5b3cad83 P 05f4ef3d126d43e9be32731e708ff76c: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/wal/wals/45d428dc499a4270b56a579f5b3cad83
I20240501 14:04:40.716130 1038 catalog_manager.cc:4830] TS 05f4ef3d126d43e9be32731e708ff76c (127.29.124.130:41511): tablet 45d428dc499a4270b56a579f5b3cad83 (table rebalance_test_table_4 [id=6fd9bb23d84f4078a694c2c7a0cb93b2]) successfully deleted
I20240501 14:04:40.743554 3428 ts_tablet_manager.cc:1875] T db2b3f73d20c4790808bbe0f1d5e489c P 5d45680f41c84ac39bd97c6f00ac2b06: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:04:40.752347 3428 ts_tablet_manager.cc:1888] T db2b3f73d20c4790808bbe0f1d5e489c P 5d45680f41c84ac39bd97c6f00ac2b06: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.103
I20240501 14:04:40.752588 3428 log.cc:1196] T db2b3f73d20c4790808bbe0f1d5e489c P 5d45680f41c84ac39bd97c6f00ac2b06: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/wal/wals/db2b3f73d20c4790808bbe0f1d5e489c
I20240501 14:04:40.756068 1036 catalog_manager.cc:4830] TS 5d45680f41c84ac39bd97c6f00ac2b06 (127.29.124.129:40925): tablet db2b3f73d20c4790808bbe0f1d5e489c (table rebalance_test_table_4 [id=6fd9bb23d84f4078a694c2c7a0cb93b2]) successfully deleted
I20240501 14:04:40.792268 1449 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6a5325b7aa824831a5fa056fb4439110"
dest_uuid: "eb004c2ac9d244f89b61448584d4115b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:56522
I20240501 14:04:40.792502 1449 raft_consensus.cc:595] T 6a5325b7aa824831a5fa056fb4439110 P eb004c2ac9d244f89b61448584d4115b [term 1 LEADER]: Received request to transfer leadership
W20240501 14:04:40.937494 1411 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56436: Service unavailable: leader transfer in progress
W20240501 14:04:40.941721 1411 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56436: Service unavailable: leader transfer in progress
W20240501 14:04:40.950321 1411 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56436: Service unavailable: leader transfer in progress
W20240501 14:04:40.960723 1411 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56436: Service unavailable: leader transfer in progress
W20240501 14:04:40.980006 1411 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56436: Service unavailable: leader transfer in progress
W20240501 14:04:40.990028 1411 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56436: Service unavailable: leader transfer in progress
W20240501 14:04:41.011420 1411 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56436: Service unavailable: leader transfer in progress
W20240501 14:04:41.035092 1411 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56436: Service unavailable: leader transfer in progress
W20240501 14:04:41.047686 1411 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56436: Service unavailable: leader transfer in progress
W20240501 14:04:41.065784 1411 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56436: Service unavailable: leader transfer in progress
W20240501 14:04:41.083262 1411 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56436: Service unavailable: leader transfer in progress
W20240501 14:04:41.102577 1413 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56436: Service unavailable: leader transfer in progress
W20240501 14:04:41.118568 1413 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56436: Service unavailable: leader transfer in progress
W20240501 14:04:41.139346 1413 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56436: Service unavailable: leader transfer in progress
W20240501 14:04:41.165467 1413 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56436: Service unavailable: leader transfer in progress
I20240501 14:04:41.201808 3418 raft_consensus.cc:979] T 6a5325b7aa824831a5fa056fb4439110 P eb004c2ac9d244f89b61448584d4115b: : Instructing follower 05f4ef3d126d43e9be32731e708ff76c to start an election
W20240501 14:04:41.206279 1413 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56436: Service unavailable: leader transfer in progress
I20240501 14:04:41.206748 3418 raft_consensus.cc:1067] T 6a5325b7aa824831a5fa056fb4439110 P eb004c2ac9d244f89b61448584d4115b [term 1 LEADER]: Signalling peer 05f4ef3d126d43e9be32731e708ff76c to start an election
I20240501 14:04:41.208385 1323 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6a5325b7aa824831a5fa056fb4439110"
dest_uuid: "05f4ef3d126d43e9be32731e708ff76c"
from {username='slave'} at 127.29.124.131:37639
I20240501 14:04:41.208662 1323 raft_consensus.cc:482] T 6a5325b7aa824831a5fa056fb4439110 P 05f4ef3d126d43e9be32731e708ff76c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:04:41.208781 1323 raft_consensus.cc:3037] T 6a5325b7aa824831a5fa056fb4439110 P 05f4ef3d126d43e9be32731e708ff76c [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:04:41.216221 1323 raft_consensus.cc:504] T 6a5325b7aa824831a5fa056fb4439110 P 05f4ef3d126d43e9be32731e708ff76c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } }
I20240501 14:04:41.219236 2487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a5325b7aa824831a5fa056fb4439110" candidate_uuid: "05f4ef3d126d43e9be32731e708ff76c" candidate_term: 2 candidate_status { last_received { term: 1 index: 103 } } ignore_live_leader: true dest_uuid: "953c332688d14831b3bc7cfd895e06e9"
I20240501 14:04:41.219568 2487 raft_consensus.cc:3037] T 6a5325b7aa824831a5fa056fb4439110 P 953c332688d14831b3bc7cfd895e06e9 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:04:41.220582 1450 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a5325b7aa824831a5fa056fb4439110" candidate_uuid: "05f4ef3d126d43e9be32731e708ff76c" candidate_term: 2 candidate_status { last_received { term: 1 index: 103 } } ignore_live_leader: true dest_uuid: "eb004c2ac9d244f89b61448584d4115b"
I20240501 14:04:41.220782 1450 raft_consensus.cc:3032] T 6a5325b7aa824831a5fa056fb4439110 P eb004c2ac9d244f89b61448584d4115b [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:04:41.220872 1450 raft_consensus.cc:729] T 6a5325b7aa824831a5fa056fb4439110 P eb004c2ac9d244f89b61448584d4115b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: eb004c2ac9d244f89b61448584d4115b, State: Running, Role: LEADER
I20240501 14:04:41.221155 1450 consensus_queue.cc:260] T 6a5325b7aa824831a5fa056fb4439110 P eb004c2ac9d244f89b61448584d4115b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 103, Committed index: 103, Last appended: 1.103, Last appended by leader: 103, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } }
I20240501 14:04:41.221547 1450 raft_consensus.cc:3037] T 6a5325b7aa824831a5fa056fb4439110 P eb004c2ac9d244f89b61448584d4115b [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:04:41.222371 2487 raft_consensus.cc:2454] T 6a5325b7aa824831a5fa056fb4439110 P 953c332688d14831b3bc7cfd895e06e9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05f4ef3d126d43e9be32731e708ff76c in term 2.
I20240501 14:04:41.224762 1258 leader_election.cc:302] T 6a5325b7aa824831a5fa056fb4439110 P 05f4ef3d126d43e9be32731e708ff76c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4ef3d126d43e9be32731e708ff76c, 953c332688d14831b3bc7cfd895e06e9; no voters:
I20240501 14:04:41.225057 3426 raft_consensus.cc:2781] T 6a5325b7aa824831a5fa056fb4439110 P 05f4ef3d126d43e9be32731e708ff76c [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:04:41.225189 3426 raft_consensus.cc:686] T 6a5325b7aa824831a5fa056fb4439110 P 05f4ef3d126d43e9be32731e708ff76c [term 2 LEADER]: Becoming Leader. State: Replica: 05f4ef3d126d43e9be32731e708ff76c, State: Running, Role: LEADER
I20240501 14:04:41.225484 3426 consensus_queue.cc:237] T 6a5325b7aa824831a5fa056fb4439110 P 05f4ef3d126d43e9be32731e708ff76c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 103, Committed index: 103, Last appended: 1.103, Last appended by leader: 103, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } }
I20240501 14:04:41.225750 1450 raft_consensus.cc:2454] T 6a5325b7aa824831a5fa056fb4439110 P eb004c2ac9d244f89b61448584d4115b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 05f4ef3d126d43e9be32731e708ff76c in term 2.
I20240501 14:04:41.228411 1052 catalog_manager.cc:5484] T 6a5325b7aa824831a5fa056fb4439110 P 05f4ef3d126d43e9be32731e708ff76c reported cstate change: term changed from 1 to 2, leader changed from eb004c2ac9d244f89b61448584d4115b (127.29.124.131) to 05f4ef3d126d43e9be32731e708ff76c (127.29.124.130). New cstate: current_term: 2 leader_uuid: "05f4ef3d126d43e9be32731e708ff76c" committed_config { opid_index: 99 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:04:41.233573 1323 leader_election.cc:288] T 6a5325b7aa824831a5fa056fb4439110 P 05f4ef3d126d43e9be32731e708ff76c [CANDIDATE]: Term 2 election: Requested vote from peers eb004c2ac9d244f89b61448584d4115b (127.29.124.131:40225), 953c332688d14831b3bc7cfd895e06e9 (127.29.124.132:34305)
W20240501 14:04:41.286371 1411 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:56436: Illegal state: replica eb004c2ac9d244f89b61448584d4115b is not leader of this config: current role FOLLOWER
I20240501 14:04:41.310621 1450 raft_consensus.cc:1261] T 6a5325b7aa824831a5fa056fb4439110 P eb004c2ac9d244f89b61448584d4115b [term 2 FOLLOWER]: Refusing update from remote peer 05f4ef3d126d43e9be32731e708ff76c: Log matching property violated. Preceding OpId in replica: term: 1 index: 103. Preceding OpId from leader: term: 2 index: 105. (index mismatch)
I20240501 14:04:41.311204 3046 consensus_queue.cc:1035] T 6a5325b7aa824831a5fa056fb4439110 P 05f4ef3d126d43e9be32731e708ff76c [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 104, Last known committed idx: 103, Time since last communication: 0.000s
I20240501 14:04:41.312206 2487 raft_consensus.cc:1261] T 6a5325b7aa824831a5fa056fb4439110 P 953c332688d14831b3bc7cfd895e06e9 [term 2 FOLLOWER]: Refusing update from remote peer 05f4ef3d126d43e9be32731e708ff76c: Log matching property violated. Preceding OpId in replica: term: 1 index: 103. Preceding OpId from leader: term: 2 index: 105. (index mismatch)
I20240501 14:04:41.313211 3046 consensus_queue.cc:1035] T 6a5325b7aa824831a5fa056fb4439110 P 05f4ef3d126d43e9be32731e708ff76c [LEADER]: Connected to new peer: Peer: permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 104, Last known committed idx: 103, Time since last communication: 0.000s
I20240501 14:04:41.395792 1193 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0f837cac32ad4aaa970124639138c2db"
dest_uuid: "5d45680f41c84ac39bd97c6f00ac2b06"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55194
I20240501 14:04:41.396082 1193 raft_consensus.cc:595] T 0f837cac32ad4aaa970124639138c2db P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 LEADER]: Received request to transfer leadership
W20240501 14:04:41.396469 3335 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55052: Service unavailable: leader transfer in progress
I20240501 14:04:41.408448 1323 consensus_queue.cc:237] T 6a5325b7aa824831a5fa056fb4439110 P 05f4ef3d126d43e9be32731e708ff76c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 105, Committed index: 105, Last appended: 2.105, Last appended by leader: 103, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } }
I20240501 14:04:41.411778 2488 raft_consensus.cc:1261] T 6a5325b7aa824831a5fa056fb4439110 P 953c332688d14831b3bc7cfd895e06e9 [term 2 FOLLOWER]: Refusing update from remote peer 05f4ef3d126d43e9be32731e708ff76c: Log matching property violated. Preceding OpId in replica: term: 2 index: 105. Preceding OpId from leader: term: 2 index: 106. (index mismatch)
W20240501 14:04:41.412308 1153 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55052: Service unavailable: leader transfer in progress
I20240501 14:04:41.412470 3427 consensus_queue.cc:1035] T 6a5325b7aa824831a5fa056fb4439110 P 05f4ef3d126d43e9be32731e708ff76c [LEADER]: Connected to new peer: Peer: permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 106, Last known committed idx: 105, Time since last communication: 0.000s
I20240501 14:04:41.416601 3426 raft_consensus.cc:2932] T 6a5325b7aa824831a5fa056fb4439110 P 05f4ef3d126d43e9be32731e708ff76c [term 2 LEADER]: Committing config change with OpId 2.106: config changed from index 99 to 106, VOTER eb004c2ac9d244f89b61448584d4115b (127.29.124.131) evicted. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } } }
W20240501 14:04:41.418758 1151 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55052: Service unavailable: leader transfer in progress
I20240501 14:04:41.427057 1052 catalog_manager.cc:5484] T 6a5325b7aa824831a5fa056fb4439110 P 05f4ef3d126d43e9be32731e708ff76c reported cstate change: config changed from index 99 to 106, VOTER eb004c2ac9d244f89b61448584d4115b (127.29.124.131) evicted. New cstate: current_term: 2 leader_uuid: "05f4ef3d126d43e9be32731e708ff76c" committed_config { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 14:04:41.430142 1151 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55052: Service unavailable: leader transfer in progress
I20240501 14:04:41.431792 1432 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a5325b7aa824831a5fa056fb4439110 with delete_type TABLET_DATA_TOMBSTONED (TS eb004c2ac9d244f89b61448584d4115b not found in new config with opid_index 106) from {username='slave'} at 127.0.0.1:38992
I20240501 14:04:41.435144 1038 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6a5325b7aa824831a5fa056fb4439110 with cas_config_opid_index 99: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:04:41.435957 2488 raft_consensus.cc:2932] T 6a5325b7aa824831a5fa056fb4439110 P 953c332688d14831b3bc7cfd895e06e9 [term 2 FOLLOWER]: Committing config change with OpId 2.106: config changed from index 99 to 106, VOTER eb004c2ac9d244f89b61448584d4115b (127.29.124.131) evicted. New config: { opid_index: 106 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } } }
I20240501 14:04:41.439723 3467 tablet_replica.cc:330] T 6a5325b7aa824831a5fa056fb4439110 P eb004c2ac9d244f89b61448584d4115b: stopping tablet replica
I20240501 14:04:41.439951 3467 raft_consensus.cc:2229] T 6a5325b7aa824831a5fa056fb4439110 P eb004c2ac9d244f89b61448584d4115b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:04:41.440248 3467 raft_consensus.cc:2258] T 6a5325b7aa824831a5fa056fb4439110 P eb004c2ac9d244f89b61448584d4115b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:04:41.441239 3467 ts_tablet_manager.cc:1875] T 6a5325b7aa824831a5fa056fb4439110 P eb004c2ac9d244f89b61448584d4115b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:04:41.448679 3467 ts_tablet_manager.cc:1888] T 6a5325b7aa824831a5fa056fb4439110 P eb004c2ac9d244f89b61448584d4115b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.105
I20240501 14:04:41.448879 3467 log.cc:1196] T 6a5325b7aa824831a5fa056fb4439110 P eb004c2ac9d244f89b61448584d4115b: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/wal/wals/6a5325b7aa824831a5fa056fb4439110
I20240501 14:04:41.449844 1039 catalog_manager.cc:4830] TS eb004c2ac9d244f89b61448584d4115b (127.29.124.131:40225): tablet 6a5325b7aa824831a5fa056fb4439110 (table rebalance_test_table_1 [id=423117c09096433e93fec4726745a428]) successfully deleted
W20240501 14:04:41.456261 1151 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55052: Service unavailable: leader transfer in progress
W20240501 14:04:41.467095 1151 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55052: Service unavailable: leader transfer in progress
W20240501 14:04:41.476841 1151 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55052: Service unavailable: leader transfer in progress
W20240501 14:04:41.491120 1151 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55052: Service unavailable: leader transfer in progress
W20240501 14:04:41.502696 1151 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55052: Service unavailable: leader transfer in progress
W20240501 14:04:41.514591 1151 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55052: Service unavailable: leader transfer in progress
W20240501 14:04:41.530081 1151 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55052: Service unavailable: leader transfer in progress
W20240501 14:04:41.543915 1151 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55052: Service unavailable: leader transfer in progress
W20240501 14:04:41.562209 1151 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55052: Service unavailable: leader transfer in progress
W20240501 14:04:41.589089 1151 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55052: Service unavailable: leader transfer in progress
W20240501 14:04:41.615868 1151 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55052: Service unavailable: leader transfer in progress
W20240501 14:04:41.663303 1151 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55052: Service unavailable: leader transfer in progress
W20240501 14:04:41.686969 1151 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55052: Service unavailable: leader transfer in progress
W20240501 14:04:41.710855 1151 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55052: Service unavailable: leader transfer in progress
W20240501 14:04:41.739848 1151 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55052: Service unavailable: leader transfer in progress
W20240501 14:04:41.763535 1151 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55052: Service unavailable: leader transfer in progress
I20240501 14:04:41.786427 3153 raft_consensus.cc:979] T 0f837cac32ad4aaa970124639138c2db P 5d45680f41c84ac39bd97c6f00ac2b06: : Instructing follower 752bc4569da647679d5df51641961840 to start an election
I20240501 14:04:41.786636 3153 raft_consensus.cc:1067] T 0f837cac32ad4aaa970124639138c2db P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 LEADER]: Signalling peer 752bc4569da647679d5df51641961840 to start an election
W20240501 14:04:41.790242 1151 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55052: Service unavailable: leader transfer in progress
I20240501 14:04:41.807595 2883 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0f837cac32ad4aaa970124639138c2db"
dest_uuid: "752bc4569da647679d5df51641961840"
from {username='slave'} at 127.29.124.129:60777
I20240501 14:04:41.807905 2883 raft_consensus.cc:482] T 0f837cac32ad4aaa970124639138c2db P 752bc4569da647679d5df51641961840 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:04:41.808033 2883 raft_consensus.cc:3037] T 0f837cac32ad4aaa970124639138c2db P 752bc4569da647679d5df51641961840 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:04:41.810945 2883 raft_consensus.cc:504] T 0f837cac32ad4aaa970124639138c2db P 752bc4569da647679d5df51641961840 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 117 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false } }
I20240501 14:04:41.815862 2883 leader_election.cc:288] T 0f837cac32ad4aaa970124639138c2db P 752bc4569da647679d5df51641961840 [CANDIDATE]: Term 2 election: Requested vote from peers eb004c2ac9d244f89b61448584d4115b (127.29.124.131:40225), 5d45680f41c84ac39bd97c6f00ac2b06 (127.29.124.129:40925)
I20240501 14:04:41.815910 1193 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f837cac32ad4aaa970124639138c2db" candidate_uuid: "752bc4569da647679d5df51641961840" candidate_term: 2 candidate_status { last_received { term: 1 index: 145 } } ignore_live_leader: true dest_uuid: "5d45680f41c84ac39bd97c6f00ac2b06"
I20240501 14:04:41.816146 1193 raft_consensus.cc:3032] T 0f837cac32ad4aaa970124639138c2db P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:04:41.816251 1193 raft_consensus.cc:729] T 0f837cac32ad4aaa970124639138c2db P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5d45680f41c84ac39bd97c6f00ac2b06, State: Running, Role: LEADER
I20240501 14:04:41.816557 1193 consensus_queue.cc:260] T 0f837cac32ad4aaa970124639138c2db P 5d45680f41c84ac39bd97c6f00ac2b06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 145, Committed index: 145, Last appended: 1.145, Last appended by leader: 145, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 117 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false } }
I20240501 14:04:41.816969 1193 raft_consensus.cc:3037] T 0f837cac32ad4aaa970124639138c2db P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:04:41.817638 1151 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55052: Illegal state: replica 5d45680f41c84ac39bd97c6f00ac2b06 is not leader of this config: current role FOLLOWER
I20240501 14:04:41.821673 1193 raft_consensus.cc:2454] T 0f837cac32ad4aaa970124639138c2db P 5d45680f41c84ac39bd97c6f00ac2b06 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 752bc4569da647679d5df51641961840 in term 2.
I20240501 14:04:41.824251 2816 leader_election.cc:302] T 0f837cac32ad4aaa970124639138c2db P 752bc4569da647679d5df51641961840 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d45680f41c84ac39bd97c6f00ac2b06, 752bc4569da647679d5df51641961840; no voters:
I20240501 14:04:41.845441 3498 raft_consensus.cc:2781] T 0f837cac32ad4aaa970124639138c2db P 752bc4569da647679d5df51641961840 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:04:41.857420 1449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f837cac32ad4aaa970124639138c2db" candidate_uuid: "752bc4569da647679d5df51641961840" candidate_term: 2 candidate_status { last_received { term: 1 index: 145 } } ignore_live_leader: true dest_uuid: "eb004c2ac9d244f89b61448584d4115b"
I20240501 14:04:41.857751 1449 raft_consensus.cc:3037] T 0f837cac32ad4aaa970124639138c2db P eb004c2ac9d244f89b61448584d4115b [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:04:41.873461 1449 raft_consensus.cc:2454] T 0f837cac32ad4aaa970124639138c2db P eb004c2ac9d244f89b61448584d4115b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 752bc4569da647679d5df51641961840 in term 2.
I20240501 14:04:41.887126 3498 raft_consensus.cc:686] T 0f837cac32ad4aaa970124639138c2db P 752bc4569da647679d5df51641961840 [term 2 LEADER]: Becoming Leader. State: Replica: 752bc4569da647679d5df51641961840, State: Running, Role: LEADER
I20240501 14:04:41.899252 3498 consensus_queue.cc:237] T 0f837cac32ad4aaa970124639138c2db P 752bc4569da647679d5df51641961840 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 145, Committed index: 145, Last appended: 1.145, Last appended by leader: 145, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 117 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false } }
W20240501 14:04:41.908708 1411 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48940: Illegal state: replica eb004c2ac9d244f89b61448584d4115b is not leader of this config: current role FOLLOWER
I20240501 14:04:41.928345 1052 catalog_manager.cc:5484] T 0f837cac32ad4aaa970124639138c2db P 752bc4569da647679d5df51641961840 reported cstate change: term changed from 1 to 2, leader changed from 5d45680f41c84ac39bd97c6f00ac2b06 (127.29.124.129) to 752bc4569da647679d5df51641961840 (127.29.124.135). New cstate: current_term: 2 leader_uuid: "752bc4569da647679d5df51641961840" committed_config { opid_index: 117 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 14:04:41.942492 1151 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55052: Illegal state: replica 5d45680f41c84ac39bd97c6f00ac2b06 is not leader of this config: current role FOLLOWER
W20240501 14:04:41.970584 1411 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:48940: Illegal state: replica eb004c2ac9d244f89b61448584d4115b is not leader of this config: current role FOLLOWER
I20240501 14:04:42.073904 1192 raft_consensus.cc:1261] T 0f837cac32ad4aaa970124639138c2db P 5d45680f41c84ac39bd97c6f00ac2b06 [term 2 FOLLOWER]: Refusing update from remote peer 752bc4569da647679d5df51641961840: Log matching property violated. Preceding OpId in replica: term: 1 index: 145. Preceding OpId from leader: term: 2 index: 147. (index mismatch)
I20240501 14:04:42.075310 1449 raft_consensus.cc:1261] T 0f837cac32ad4aaa970124639138c2db P eb004c2ac9d244f89b61448584d4115b [term 2 FOLLOWER]: Refusing update from remote peer 752bc4569da647679d5df51641961840: Log matching property violated. Preceding OpId in replica: term: 1 index: 145. Preceding OpId from leader: term: 2 index: 147. (index mismatch)
I20240501 14:04:42.075510 3498 consensus_queue.cc:1035] T 0f837cac32ad4aaa970124639138c2db P 752bc4569da647679d5df51641961840 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 146, Last known committed idx: 145, Time since last communication: 0.000s
I20240501 14:04:42.079191 3498 consensus_queue.cc:1035] T 0f837cac32ad4aaa970124639138c2db P 752bc4569da647679d5df51641961840 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 146, Last known committed idx: 145, Time since last communication: 0.000s
I20240501 14:04:42.088572 3397 mvcc.cc:205] Tried to move back new op lower bound from 7022888067363704832 to 7022888066700611584. Current Snapshot: MvccSnapshot[applied={T|T < 7022888067363704832}]
I20240501 14:04:42.249648 2884 consensus_queue.cc:237] T 0f837cac32ad4aaa970124639138c2db P 752bc4569da647679d5df51641961840 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 150, Committed index: 150, Last appended: 2.151, Last appended by leader: 145, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 152 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false } }
I20240501 14:04:42.251965 1453 raft_consensus.cc:1261] T 0f837cac32ad4aaa970124639138c2db P eb004c2ac9d244f89b61448584d4115b [term 2 FOLLOWER]: Refusing update from remote peer 752bc4569da647679d5df51641961840: Log matching property violated. Preceding OpId in replica: term: 2 index: 151. Preceding OpId from leader: term: 2 index: 152. (index mismatch)
I20240501 14:04:42.253849 3498 consensus_queue.cc:1035] T 0f837cac32ad4aaa970124639138c2db P 752bc4569da647679d5df51641961840 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 152, Last known committed idx: 151, Time since last communication: 0.000s
I20240501 14:04:42.257038 3498 raft_consensus.cc:2932] T 0f837cac32ad4aaa970124639138c2db P 752bc4569da647679d5df51641961840 [term 2 LEADER]: Committing config change with OpId 2.152: config changed from index 117 to 152, VOTER 5d45680f41c84ac39bd97c6f00ac2b06 (127.29.124.129) evicted. New config: { opid_index: 152 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false } } }
I20240501 14:04:42.259718 1453 raft_consensus.cc:2932] T 0f837cac32ad4aaa970124639138c2db P eb004c2ac9d244f89b61448584d4115b [term 2 FOLLOWER]: Committing config change with OpId 2.152: config changed from index 117 to 152, VOTER 5d45680f41c84ac39bd97c6f00ac2b06 (127.29.124.129) evicted. New config: { opid_index: 152 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false } } }
I20240501 14:04:42.260756 1038 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0f837cac32ad4aaa970124639138c2db with cas_config_opid_index 117: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:04:42.262663 1052 catalog_manager.cc:5484] T 0f837cac32ad4aaa970124639138c2db P 752bc4569da647679d5df51641961840 reported cstate change: config changed from index 117 to 152, VOTER 5d45680f41c84ac39bd97c6f00ac2b06 (127.29.124.129) evicted. New cstate: current_term: 2 leader_uuid: "752bc4569da647679d5df51641961840" committed_config { opid_index: 152 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:04:42.267783 1172 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f837cac32ad4aaa970124639138c2db with delete_type TABLET_DATA_TOMBSTONED (TS 5d45680f41c84ac39bd97c6f00ac2b06 not found in new config with opid_index 152) from {username='slave'} at 127.0.0.1:51144
I20240501 14:04:42.268414 3529 tablet_replica.cc:330] T 0f837cac32ad4aaa970124639138c2db P 5d45680f41c84ac39bd97c6f00ac2b06: stopping tablet replica
I20240501 14:04:42.268658 3529 raft_consensus.cc:2229] T 0f837cac32ad4aaa970124639138c2db P 5d45680f41c84ac39bd97c6f00ac2b06 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:04:42.268877 3529 pending_rounds.cc:62] T 0f837cac32ad4aaa970124639138c2db P 5d45680f41c84ac39bd97c6f00ac2b06: Trying to abort 1 pending ops.
I20240501 14:04:42.269007 3529 pending_rounds.cc:68] T 0f837cac32ad4aaa970124639138c2db P 5d45680f41c84ac39bd97c6f00ac2b06: Aborting op as it isn't in flight: id { term: 2 index: 151 } timestamp: 7022888068077236224 op_type: WRITE_OP write_request { tablet_id: "0f837cac32ad4aaa970124639138c2db" schema { columns { name: "key" type: INT32 is_key: true is_nullable: false immutable: false } columns { name: "int_val" type: INT32 is_key: false is_nullable: false immutable: false } columns { name: "string_val" type: STRING is_key: false is_nullable: true immutable: false } } row_operations { rows: "<redacted>""\001\007\000\346\364\027n\342\351\310\270\000\000\000\000\000\000\000\000\020\000\000\000\000\000\000\000" indirect_data: "<redacted>""9242c485bae843e8" } external_consistency_mode: CLIENT_PROPAGATED propagated_timestamp: 7022888068049981440 authz_token { token_data: "\010\236\232\311\261\006\"3\n\005slave\022*\n 817d8db2cf8f4cffb29e21f78e86e2d5\020\001\030\001 \001(\001" signature: "<redacted>""\322V,4\241\212\365G\301\003\366^{?7\335\344\002t\242\351\264HDt\261\233{\376\375\"\243B\247SntX\246\231\217\314K?~t\206D\301d\214Mft\323\312Z\315\003\314\345%C\314" signing_key_seq_num: 0 } } request_id { client_id: "770c0c789dd64fe394cefb7630187f21" seq_no: 431 first_incomplete_seq_no: 431 attempt_no: 0 }
I20240501 14:04:42.269800 3529 raft_consensus.cc:2258] T 0f837cac32ad4aaa970124639138c2db P 5d45680f41c84ac39bd97c6f00ac2b06 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:04:42.270869 3529 ts_tablet_manager.cc:1875] T 0f837cac32ad4aaa970124639138c2db P 5d45680f41c84ac39bd97c6f00ac2b06: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:04:42.277840 3529 ts_tablet_manager.cc:1888] T 0f837cac32ad4aaa970124639138c2db P 5d45680f41c84ac39bd97c6f00ac2b06: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.151
I20240501 14:04:42.278060 3529 log.cc:1196] T 0f837cac32ad4aaa970124639138c2db P 5d45680f41c84ac39bd97c6f00ac2b06: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/wal/wals/0f837cac32ad4aaa970124639138c2db
I20240501 14:04:42.279314 1036 catalog_manager.cc:4830] TS 5d45680f41c84ac39bd97c6f00ac2b06 (127.29.124.129:40925): tablet 0f837cac32ad4aaa970124639138c2db (table rebalance_test_table_0 [id=817d8db2cf8f4cffb29e21f78e86e2d5]) successfully deleted
I20240501 14:04:42.360416 1323 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "53249c71dab94ff4b179309088f4d1cd"
dest_uuid: "05f4ef3d126d43e9be32731e708ff76c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52194
I20240501 14:04:42.360695 1323 raft_consensus.cc:595] T 53249c71dab94ff4b179309088f4d1cd P 05f4ef3d126d43e9be32731e708ff76c [term 1 LEADER]: Received request to transfer leadership
W20240501 14:04:42.398715 1283 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58204: Service unavailable: leader transfer in progress
W20240501 14:04:42.405486 1283 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58204: Service unavailable: leader transfer in progress
W20240501 14:04:42.410241 1283 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58204: Service unavailable: leader transfer in progress
W20240501 14:04:42.416201 1282 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58204: Service unavailable: leader transfer in progress
W20240501 14:04:42.457661 1282 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58204: Service unavailable: leader transfer in progress
W20240501 14:04:42.474284 1283 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58204: Service unavailable: leader transfer in progress
W20240501 14:04:42.501996 1282 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58204: Service unavailable: leader transfer in progress
W20240501 14:04:42.517701 1283 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58204: Service unavailable: leader transfer in progress
W20240501 14:04:42.538419 1283 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58204: Service unavailable: leader transfer in progress
W20240501 14:04:42.553972 1282 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58204: Service unavailable: leader transfer in progress
W20240501 14:04:42.571542 1282 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58204: Service unavailable: leader transfer in progress
W20240501 14:04:42.585597 1282 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58204: Service unavailable: leader transfer in progress
W20240501 14:04:42.599828 1282 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58204: Service unavailable: leader transfer in progress
W20240501 14:04:42.634166 1282 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58204: Service unavailable: leader transfer in progress
W20240501 14:04:42.652982 1282 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58204: Service unavailable: leader transfer in progress
W20240501 14:04:42.676076 1282 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58204: Service unavailable: leader transfer in progress
W20240501 14:04:42.708926 1281 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58204: Service unavailable: leader transfer in progress
I20240501 14:04:42.711612 3519 raft_consensus.cc:979] T 53249c71dab94ff4b179309088f4d1cd P 05f4ef3d126d43e9be32731e708ff76c: : Instructing follower eb004c2ac9d244f89b61448584d4115b to start an election
I20240501 14:04:42.711787 3519 raft_consensus.cc:1067] T 53249c71dab94ff4b179309088f4d1cd P 05f4ef3d126d43e9be32731e708ff76c [term 1 LEADER]: Signalling peer eb004c2ac9d244f89b61448584d4115b to start an election
I20240501 14:04:42.712790 1453 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "53249c71dab94ff4b179309088f4d1cd"
dest_uuid: "eb004c2ac9d244f89b61448584d4115b"
from {username='slave'} at 127.29.124.130:47957
I20240501 14:04:42.713032 1453 raft_consensus.cc:482] T 53249c71dab94ff4b179309088f4d1cd P eb004c2ac9d244f89b61448584d4115b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:04:42.713182 1453 raft_consensus.cc:3037] T 53249c71dab94ff4b179309088f4d1cd P eb004c2ac9d244f89b61448584d4115b [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:04:42.716107 1453 raft_consensus.cc:504] T 53249c71dab94ff4b179309088f4d1cd P eb004c2ac9d244f89b61448584d4115b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 101 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } }
I20240501 14:04:42.717036 1453 leader_election.cc:288] T 53249c71dab94ff4b179309088f4d1cd P eb004c2ac9d244f89b61448584d4115b [CANDIDATE]: Term 2 election: Requested vote from peers 05f4ef3d126d43e9be32731e708ff76c (127.29.124.130:41511), 953c332688d14831b3bc7cfd895e06e9 (127.29.124.132:34305)
I20240501 14:04:42.718621 1323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53249c71dab94ff4b179309088f4d1cd" candidate_uuid: "eb004c2ac9d244f89b61448584d4115b" candidate_term: 2 candidate_status { last_received { term: 1 index: 146 } } ignore_live_leader: true dest_uuid: "05f4ef3d126d43e9be32731e708ff76c"
I20240501 14:04:42.718856 1323 raft_consensus.cc:3032] T 53249c71dab94ff4b179309088f4d1cd P 05f4ef3d126d43e9be32731e708ff76c [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:04:42.718952 1323 raft_consensus.cc:729] T 53249c71dab94ff4b179309088f4d1cd P 05f4ef3d126d43e9be32731e708ff76c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 05f4ef3d126d43e9be32731e708ff76c, State: Running, Role: LEADER
I20240501 14:04:42.719282 1323 consensus_queue.cc:260] T 53249c71dab94ff4b179309088f4d1cd P 05f4ef3d126d43e9be32731e708ff76c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 146, Committed index: 146, Last appended: 1.146, Last appended by leader: 146, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 101 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } }
I20240501 14:04:42.719648 1323 raft_consensus.cc:3037] T 53249c71dab94ff4b179309088f4d1cd P 05f4ef3d126d43e9be32731e708ff76c [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:04:42.722004 2486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53249c71dab94ff4b179309088f4d1cd" candidate_uuid: "eb004c2ac9d244f89b61448584d4115b" candidate_term: 2 candidate_status { last_received { term: 1 index: 146 } } ignore_live_leader: true dest_uuid: "953c332688d14831b3bc7cfd895e06e9"
I20240501 14:04:42.722208 2486 raft_consensus.cc:3037] T 53249c71dab94ff4b179309088f4d1cd P 953c332688d14831b3bc7cfd895e06e9 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:04:42.727162 1323 raft_consensus.cc:2454] T 53249c71dab94ff4b179309088f4d1cd P 05f4ef3d126d43e9be32731e708ff76c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb004c2ac9d244f89b61448584d4115b in term 2.
I20240501 14:04:42.727720 1388 leader_election.cc:302] T 53249c71dab94ff4b179309088f4d1cd P eb004c2ac9d244f89b61448584d4115b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4ef3d126d43e9be32731e708ff76c, eb004c2ac9d244f89b61448584d4115b; no voters:
I20240501 14:04:42.728755 3567 raft_consensus.cc:2781] T 53249c71dab94ff4b179309088f4d1cd P eb004c2ac9d244f89b61448584d4115b [term 2 FOLLOWER]: Leader election won for term 2
W20240501 14:04:42.729588 1282 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58204: Illegal state: replica 05f4ef3d126d43e9be32731e708ff76c is not leader of this config: current role FOLLOWER
I20240501 14:04:42.731356 3567 raft_consensus.cc:686] T 53249c71dab94ff4b179309088f4d1cd P eb004c2ac9d244f89b61448584d4115b [term 2 LEADER]: Becoming Leader. State: Replica: eb004c2ac9d244f89b61448584d4115b, State: Running, Role: LEADER
I20240501 14:04:42.732271 3567 consensus_queue.cc:237] T 53249c71dab94ff4b179309088f4d1cd P eb004c2ac9d244f89b61448584d4115b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 146, Committed index: 146, Last appended: 1.146, Last appended by leader: 146, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 101 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } }
I20240501 14:04:42.735424 1052 catalog_manager.cc:5484] T 53249c71dab94ff4b179309088f4d1cd P eb004c2ac9d244f89b61448584d4115b reported cstate change: term changed from 1 to 2, leader changed from 05f4ef3d126d43e9be32731e708ff76c (127.29.124.130) to eb004c2ac9d244f89b61448584d4115b (127.29.124.131). New cstate: current_term: 2 leader_uuid: "eb004c2ac9d244f89b61448584d4115b" committed_config { opid_index: 101 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:04:42.743427 2486 raft_consensus.cc:2454] T 53249c71dab94ff4b179309088f4d1cd P 953c332688d14831b3bc7cfd895e06e9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb004c2ac9d244f89b61448584d4115b in term 2.
I20240501 14:04:42.828552 1323 raft_consensus.cc:1261] T 53249c71dab94ff4b179309088f4d1cd P 05f4ef3d126d43e9be32731e708ff76c [term 2 FOLLOWER]: Refusing update from remote peer eb004c2ac9d244f89b61448584d4115b: Log matching property violated. Preceding OpId in replica: term: 1 index: 146. Preceding OpId from leader: term: 2 index: 148. (index mismatch)
I20240501 14:04:42.829500 2486 raft_consensus.cc:1261] T 53249c71dab94ff4b179309088f4d1cd P 953c332688d14831b3bc7cfd895e06e9 [term 2 FOLLOWER]: Refusing update from remote peer eb004c2ac9d244f89b61448584d4115b: Log matching property violated. Preceding OpId in replica: term: 1 index: 146. Preceding OpId from leader: term: 2 index: 148. (index mismatch)
I20240501 14:04:42.830206 3566 consensus_queue.cc:1035] T 53249c71dab94ff4b179309088f4d1cd P eb004c2ac9d244f89b61448584d4115b [LEADER]: Connected to new peer: Peer: permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 147, Last known committed idx: 146, Time since last communication: 0.000s
I20240501 14:04:42.830753 3418 consensus_queue.cc:1035] T 53249c71dab94ff4b179309088f4d1cd P eb004c2ac9d244f89b61448584d4115b [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 147, Last known committed idx: 146, Time since last communication: 0.000s
I20240501 14:04:42.835381 3401 mvcc.cc:205] Tried to move back new op lower bound from 7022888070453194752 to 7022888070075060224. Current Snapshot: MvccSnapshot[applied={T|T < 7022888070453194752}]
I20240501 14:04:42.841616 1449 consensus_queue.cc:237] T 53249c71dab94ff4b179309088f4d1cd P eb004c2ac9d244f89b61448584d4115b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 148, Committed index: 148, Last appended: 2.149, Last appended by leader: 146, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 150 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } }
I20240501 14:04:42.843956 2486 raft_consensus.cc:1261] T 53249c71dab94ff4b179309088f4d1cd P 953c332688d14831b3bc7cfd895e06e9 [term 2 FOLLOWER]: Refusing update from remote peer eb004c2ac9d244f89b61448584d4115b: Log matching property violated. Preceding OpId in replica: term: 2 index: 148. Preceding OpId from leader: term: 2 index: 150. (index mismatch)
I20240501 14:04:42.847280 3566 consensus_queue.cc:1035] T 53249c71dab94ff4b179309088f4d1cd P eb004c2ac9d244f89b61448584d4115b [LEADER]: Connected to new peer: Peer: permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 150, Last known committed idx: 148, Time since last communication: 0.000s
I20240501 14:04:42.851563 3418 raft_consensus.cc:2932] T 53249c71dab94ff4b179309088f4d1cd P eb004c2ac9d244f89b61448584d4115b [term 2 LEADER]: Committing config change with OpId 2.150: config changed from index 101 to 150, VOTER 05f4ef3d126d43e9be32731e708ff76c (127.29.124.130) evicted. New config: { opid_index: 150 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } } }
I20240501 14:04:42.855363 1039 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 53249c71dab94ff4b179309088f4d1cd with cas_config_opid_index 101: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:04:42.857663 2487 raft_consensus.cc:2932] T 53249c71dab94ff4b179309088f4d1cd P 953c332688d14831b3bc7cfd895e06e9 [term 2 FOLLOWER]: Committing config change with OpId 2.150: config changed from index 101 to 150, VOTER 05f4ef3d126d43e9be32731e708ff76c (127.29.124.130) evicted. New config: { opid_index: 150 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } } }
I20240501 14:04:42.858793 1052 catalog_manager.cc:5484] T 53249c71dab94ff4b179309088f4d1cd P eb004c2ac9d244f89b61448584d4115b reported cstate change: config changed from index 101 to 150, VOTER 05f4ef3d126d43e9be32731e708ff76c (127.29.124.130) evicted. New cstate: current_term: 2 leader_uuid: "eb004c2ac9d244f89b61448584d4115b" committed_config { opid_index: 150 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:04:42.866775 1302 tablet_service.cc:1514] Processing DeleteTablet for tablet 53249c71dab94ff4b179309088f4d1cd with delete_type TABLET_DATA_TOMBSTONED (TS 05f4ef3d126d43e9be32731e708ff76c not found in new config with opid_index 150) from {username='slave'} at 127.0.0.1:35916
I20240501 14:04:42.871266 3574 tablet_replica.cc:330] T 53249c71dab94ff4b179309088f4d1cd P 05f4ef3d126d43e9be32731e708ff76c: stopping tablet replica
I20240501 14:04:42.871701 3574 raft_consensus.cc:2229] T 53249c71dab94ff4b179309088f4d1cd P 05f4ef3d126d43e9be32731e708ff76c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:04:42.872182 3574 raft_consensus.cc:2258] T 53249c71dab94ff4b179309088f4d1cd P 05f4ef3d126d43e9be32731e708ff76c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:04:42.875272 3574 ts_tablet_manager.cc:1875] T 53249c71dab94ff4b179309088f4d1cd P 05f4ef3d126d43e9be32731e708ff76c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:04:42.890265 3574 ts_tablet_manager.cc:1888] T 53249c71dab94ff4b179309088f4d1cd P 05f4ef3d126d43e9be32731e708ff76c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.148
I20240501 14:04:42.890509 3574 log.cc:1196] T 53249c71dab94ff4b179309088f4d1cd P 05f4ef3d126d43e9be32731e708ff76c: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/wal/wals/53249c71dab94ff4b179309088f4d1cd
I20240501 14:04:42.892307 1038 catalog_manager.cc:4830] TS 05f4ef3d126d43e9be32731e708ff76c (127.29.124.130:41511): tablet 53249c71dab94ff4b179309088f4d1cd (table rebalance_test_table_2 [id=9c1d190b2b264f27af23180ab8f30d42]) successfully deleted
I20240501 14:04:43.729434 2733 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:04:43.784408 1302 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:04:43.790393 2602 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:04:43.824420 2864 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:04:43.825210 2469 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:04:43.833232 1172 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:43.852459 1432 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:04:44.298849 1193 consensus_queue.cc:237] T 0f11914838ca4139a1c1813fc5334782 P 5d45680f41c84ac39bd97c6f00ac2b06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 195, Committed index: 195, Last appended: 1.195, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 196 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: true } }
I20240501 14:04:44.306311 2622 raft_consensus.cc:1261] T 0f11914838ca4139a1c1813fc5334782 P 4e6923f9753a4450ac4343c34c427d1e [term 1 FOLLOWER]: Refusing update from remote peer 5d45680f41c84ac39bd97c6f00ac2b06: Log matching property violated. Preceding OpId in replica: term: 1 index: 195. Preceding OpId from leader: term: 1 index: 196. (index mismatch)
W20240501 14:04:44.306663 1128 consensus_peers.cc:485] T 0f11914838ca4139a1c1813fc5334782 P 5d45680f41c84ac39bd97c6f00ac2b06 -> Peer 752bc4569da647679d5df51641961840 (127.29.124.135:42923): Couldn't send request to peer 752bc4569da647679d5df51641961840. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f11914838ca4139a1c1813fc5334782. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:04:44.307761 3599 consensus_queue.cc:1035] T 0f11914838ca4139a1c1813fc5334782 P 5d45680f41c84ac39bd97c6f00ac2b06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 196, Last known committed idx: 195, Time since last communication: 0.000s
I20240501 14:04:44.311125 3599 raft_consensus.cc:2932] T 0f11914838ca4139a1c1813fc5334782 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 LEADER]: Committing config change with OpId 1.196: config changed from index 80 to 196, NON_VOTER 752bc4569da647679d5df51641961840 (127.29.124.135) added. New config: { opid_index: 196 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: true } } }
I20240501 14:04:44.317018 1045 catalog_manager.cc:5484] T 0f11914838ca4139a1c1813fc5334782 P 5d45680f41c84ac39bd97c6f00ac2b06 reported cstate change: config changed from index 80 to 196, NON_VOTER 752bc4569da647679d5df51641961840 (127.29.124.135) added. New cstate: current_term: 1 leader_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" committed_config { opid_index: 196 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:04:44.315963 2622 raft_consensus.cc:2932] T 0f11914838ca4139a1c1813fc5334782 P 4e6923f9753a4450ac4343c34c427d1e [term 1 FOLLOWER]: Committing config change with OpId 1.196: config changed from index 80 to 196, NON_VOTER 752bc4569da647679d5df51641961840 (127.29.124.135) added. New config: { opid_index: 196 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: true } } }
I20240501 14:04:44.469708 1323 consensus_queue.cc:237] T 6a5325b7aa824831a5fa056fb4439110 P 05f4ef3d126d43e9be32731e708ff76c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 177, Committed index: 177, Last appended: 2.177, Last appended by leader: 103, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 178 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } } peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: true } }
I20240501 14:04:44.480013 2486 raft_consensus.cc:1261] T 6a5325b7aa824831a5fa056fb4439110 P 953c332688d14831b3bc7cfd895e06e9 [term 2 FOLLOWER]: Refusing update from remote peer 05f4ef3d126d43e9be32731e708ff76c: Log matching property violated. Preceding OpId in replica: term: 2 index: 177. Preceding OpId from leader: term: 2 index: 178. (index mismatch)
I20240501 14:04:44.482453 3427 consensus_queue.cc:1035] T 6a5325b7aa824831a5fa056fb4439110 P 05f4ef3d126d43e9be32731e708ff76c [LEADER]: Connected to new peer: Peer: permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 178, Last known committed idx: 177, Time since last communication: 0.000s
I20240501 14:04:44.487190 3542 raft_consensus.cc:2932] T 6a5325b7aa824831a5fa056fb4439110 P 05f4ef3d126d43e9be32731e708ff76c [term 2 LEADER]: Committing config change with OpId 2.178: config changed from index 106 to 178, NON_VOTER 4e6923f9753a4450ac4343c34c427d1e (127.29.124.133) added. New config: { opid_index: 178 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } } peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: true } } }
I20240501 14:04:44.487917 2486 raft_consensus.cc:2932] T 6a5325b7aa824831a5fa056fb4439110 P 953c332688d14831b3bc7cfd895e06e9 [term 2 FOLLOWER]: Committing config change with OpId 2.178: config changed from index 106 to 178, NON_VOTER 4e6923f9753a4450ac4343c34c427d1e (127.29.124.133) added. New config: { opid_index: 178 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } } peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: true } } }
I20240501 14:04:44.494639 1050 catalog_manager.cc:5484] T 6a5325b7aa824831a5fa056fb4439110 P 05f4ef3d126d43e9be32731e708ff76c reported cstate change: config changed from index 106 to 178, NON_VOTER 4e6923f9753a4450ac4343c34c427d1e (127.29.124.133) added. New cstate: current_term: 2 leader_uuid: "05f4ef3d126d43e9be32731e708ff76c" committed_config { opid_index: 178 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 14:04:44.529106 1256 consensus_peers.cc:485] T 6a5325b7aa824831a5fa056fb4439110 P 05f4ef3d126d43e9be32731e708ff76c -> Peer 4e6923f9753a4450ac4343c34c427d1e (127.29.124.133:34675): Couldn't send request to peer 4e6923f9753a4450ac4343c34c427d1e. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6a5325b7aa824831a5fa056fb4439110. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:04:44.902426 3667 ts_tablet_manager.cc:902] T 6a5325b7aa824831a5fa056fb4439110 P 4e6923f9753a4450ac4343c34c427d1e: Initiating tablet copy from peer 05f4ef3d126d43e9be32731e708ff76c (127.29.124.130:41511)
I20240501 14:04:44.926728 3669 ts_tablet_manager.cc:902] T 0f11914838ca4139a1c1813fc5334782 P 752bc4569da647679d5df51641961840: Initiating tablet copy from peer 5d45680f41c84ac39bd97c6f00ac2b06 (127.29.124.129:40925)
I20240501 14:04:44.999089 3667 tablet_copy_client.cc:308] T 6a5325b7aa824831a5fa056fb4439110 P 4e6923f9753a4450ac4343c34c427d1e: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.130:41511
I20240501 14:04:45.010489 3669 tablet_copy_client.cc:308] T 0f11914838ca4139a1c1813fc5334782 P 752bc4569da647679d5df51641961840: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.129:40925
I20240501 14:04:45.014940 1213 tablet_copy_service.cc:140] P 5d45680f41c84ac39bd97c6f00ac2b06: Received BeginTabletCopySession request for tablet 0f11914838ca4139a1c1813fc5334782 from peer 752bc4569da647679d5df51641961840 ({username='slave'} at 127.29.124.135:47221)
I20240501 14:04:45.015094 1213 tablet_copy_service.cc:161] P 5d45680f41c84ac39bd97c6f00ac2b06: Beginning new tablet copy session on tablet 0f11914838ca4139a1c1813fc5334782 from peer 752bc4569da647679d5df51641961840 at {username='slave'} at 127.29.124.135:47221: session id = 752bc4569da647679d5df51641961840-0f11914838ca4139a1c1813fc5334782
I20240501 14:04:45.017735 1213 tablet_copy_source_session.cc:205] T 0f11914838ca4139a1c1813fc5334782 P 5d45680f41c84ac39bd97c6f00ac2b06: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:04:45.025229 1343 tablet_copy_service.cc:140] P 05f4ef3d126d43e9be32731e708ff76c: Received BeginTabletCopySession request for tablet 6a5325b7aa824831a5fa056fb4439110 from peer 4e6923f9753a4450ac4343c34c427d1e ({username='slave'} at 127.29.124.133:43219)
I20240501 14:04:45.025408 1343 tablet_copy_service.cc:161] P 05f4ef3d126d43e9be32731e708ff76c: Beginning new tablet copy session on tablet 6a5325b7aa824831a5fa056fb4439110 from peer 4e6923f9753a4450ac4343c34c427d1e at {username='slave'} at 127.29.124.133:43219: session id = 4e6923f9753a4450ac4343c34c427d1e-6a5325b7aa824831a5fa056fb4439110
I20240501 14:04:45.027699 3669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f11914838ca4139a1c1813fc5334782. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:45.028050 1343 tablet_copy_source_session.cc:205] T 6a5325b7aa824831a5fa056fb4439110 P 05f4ef3d126d43e9be32731e708ff76c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:04:45.031277 3667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a5325b7aa824831a5fa056fb4439110. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:45.063954 3667 tablet_copy_client.cc:783] T 6a5325b7aa824831a5fa056fb4439110 P 4e6923f9753a4450ac4343c34c427d1e: tablet copy: Starting download of 0 data blocks...
I20240501 14:04:45.064575 3669 tablet_copy_client.cc:783] T 0f11914838ca4139a1c1813fc5334782 P 752bc4569da647679d5df51641961840: tablet copy: Starting download of 0 data blocks...
I20240501 14:04:45.064859 3667 tablet_copy_client.cc:647] T 6a5325b7aa824831a5fa056fb4439110 P 4e6923f9753a4450ac4343c34c427d1e: tablet copy: Starting download of 1 WAL segments...
I20240501 14:04:45.065138 3669 tablet_copy_client.cc:647] T 0f11914838ca4139a1c1813fc5334782 P 752bc4569da647679d5df51641961840: tablet copy: Starting download of 1 WAL segments...
I20240501 14:04:45.067726 3669 tablet_copy_client.cc:523] T 0f11914838ca4139a1c1813fc5334782 P 752bc4569da647679d5df51641961840: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:04:45.067957 3667 tablet_copy_client.cc:523] T 6a5325b7aa824831a5fa056fb4439110 P 4e6923f9753a4450ac4343c34c427d1e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:04:45.071259 3669 tablet_bootstrap.cc:492] T 0f11914838ca4139a1c1813fc5334782 P 752bc4569da647679d5df51641961840: Bootstrap starting.
I20240501 14:04:45.082396 3667 tablet_bootstrap.cc:492] T 6a5325b7aa824831a5fa056fb4439110 P 4e6923f9753a4450ac4343c34c427d1e: Bootstrap starting.
I20240501 14:04:45.389737 3669 tablet_bootstrap.cc:492] T 0f11914838ca4139a1c1813fc5334782 P 752bc4569da647679d5df51641961840: Bootstrap replayed 1/1 log segments. Stats: ops{read=209 overwritten=0 applied=208 ignored=0} inserts{seen=836 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 14:04:45.390355 3669 tablet_bootstrap.cc:492] T 0f11914838ca4139a1c1813fc5334782 P 752bc4569da647679d5df51641961840: Bootstrap complete.
I20240501 14:04:45.390735 3669 ts_tablet_manager.cc:1367] T 0f11914838ca4139a1c1813fc5334782 P 752bc4569da647679d5df51641961840: Time spent bootstrapping tablet: real 0.320s user 0.248s sys 0.038s
I20240501 14:04:45.391561 3669 raft_consensus.cc:348] T 0f11914838ca4139a1c1813fc5334782 P 752bc4569da647679d5df51641961840 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 196 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: true } }
I20240501 14:04:45.392635 3669 raft_consensus.cc:729] T 0f11914838ca4139a1c1813fc5334782 P 752bc4569da647679d5df51641961840 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 752bc4569da647679d5df51641961840, State: Initialized, Role: LEARNER
I20240501 14:04:45.392899 3669 consensus_queue.cc:260] T 0f11914838ca4139a1c1813fc5334782 P 752bc4569da647679d5df51641961840 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 208, Last appended: 1.209, Last appended by leader: 209, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 196 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: true } }
I20240501 14:04:45.393576 3669 ts_tablet_manager.cc:1398] T 0f11914838ca4139a1c1813fc5334782 P 752bc4569da647679d5df51641961840: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:04:45.404060 1213 tablet_copy_service.cc:342] P 5d45680f41c84ac39bd97c6f00ac2b06: Request end of tablet copy session 752bc4569da647679d5df51641961840-0f11914838ca4139a1c1813fc5334782 received from {username='slave'} at 127.29.124.135:47221
I20240501 14:04:45.404271 1213 tablet_copy_service.cc:433] P 5d45680f41c84ac39bd97c6f00ac2b06: ending tablet copy session 752bc4569da647679d5df51641961840-0f11914838ca4139a1c1813fc5334782 on tablet 0f11914838ca4139a1c1813fc5334782 with peer 752bc4569da647679d5df51641961840
I20240501 14:04:45.509079 2884 raft_consensus.cc:1203] T 0f11914838ca4139a1c1813fc5334782 P 752bc4569da647679d5df51641961840 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.195->[1.196-1.220] Dedup: 1.209->[1.210-1.220]
I20240501 14:04:45.587803 3066 raft_consensus.cc:1050] T 0f11914838ca4139a1c1813fc5334782 P 5d45680f41c84ac39bd97c6f00ac2b06: attempting to promote NON_VOTER 752bc4569da647679d5df51641961840 to VOTER
I20240501 14:04:45.588485 3066 consensus_queue.cc:237] T 0f11914838ca4139a1c1813fc5334782 P 5d45680f41c84ac39bd97c6f00ac2b06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 223, Committed index: 223, Last appended: 1.223, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 224 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false } }
I20240501 14:04:45.590500 2621 raft_consensus.cc:1261] T 0f11914838ca4139a1c1813fc5334782 P 4e6923f9753a4450ac4343c34c427d1e [term 1 FOLLOWER]: Refusing update from remote peer 5d45680f41c84ac39bd97c6f00ac2b06: Log matching property violated. Preceding OpId in replica: term: 1 index: 223. Preceding OpId from leader: term: 1 index: 224. (index mismatch)
I20240501 14:04:45.590546 2884 raft_consensus.cc:1261] T 0f11914838ca4139a1c1813fc5334782 P 752bc4569da647679d5df51641961840 [term 1 LEARNER]: Refusing update from remote peer 5d45680f41c84ac39bd97c6f00ac2b06: Log matching property violated. Preceding OpId in replica: term: 1 index: 223. Preceding OpId from leader: term: 1 index: 224. (index mismatch)
I20240501 14:04:45.590975 3066 consensus_queue.cc:1035] T 0f11914838ca4139a1c1813fc5334782 P 5d45680f41c84ac39bd97c6f00ac2b06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 224, Last known committed idx: 223, Time since last communication: 0.000s
I20240501 14:04:45.593551 3066 consensus_queue.cc:1035] T 0f11914838ca4139a1c1813fc5334782 P 5d45680f41c84ac39bd97c6f00ac2b06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 224, Last known committed idx: 223, Time since last communication: 0.000s
I20240501 14:04:45.596071 3667 tablet_bootstrap.cc:492] T 6a5325b7aa824831a5fa056fb4439110 P 4e6923f9753a4450ac4343c34c427d1e: Bootstrap replayed 1/1 log segments. Stats: ops{read=187 overwritten=0 applied=186 ignored=0} inserts{seen=854 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 14:04:45.597149 3153 raft_consensus.cc:2932] T 0f11914838ca4139a1c1813fc5334782 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 LEADER]: Committing config change with OpId 1.224: config changed from index 196 to 224, 752bc4569da647679d5df51641961840 (127.29.124.135) changed from NON_VOTER to VOTER. New config: { opid_index: 224 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false } } }
I20240501 14:04:45.604645 3667 tablet_bootstrap.cc:492] T 6a5325b7aa824831a5fa056fb4439110 P 4e6923f9753a4450ac4343c34c427d1e: Bootstrap complete.
I20240501 14:04:45.605381 3667 ts_tablet_manager.cc:1367] T 6a5325b7aa824831a5fa056fb4439110 P 4e6923f9753a4450ac4343c34c427d1e: Time spent bootstrapping tablet: real 0.523s user 0.247s sys 0.028s
I20240501 14:04:45.606189 2883 raft_consensus.cc:2932] T 0f11914838ca4139a1c1813fc5334782 P 752bc4569da647679d5df51641961840 [term 1 FOLLOWER]: Committing config change with OpId 1.224: config changed from index 196 to 224, 752bc4569da647679d5df51641961840 (127.29.124.135) changed from NON_VOTER to VOTER. New config: { opid_index: 224 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false } } }
I20240501 14:04:45.604893 2621 raft_consensus.cc:2932] T 0f11914838ca4139a1c1813fc5334782 P 4e6923f9753a4450ac4343c34c427d1e [term 1 FOLLOWER]: Committing config change with OpId 1.224: config changed from index 196 to 224, 752bc4569da647679d5df51641961840 (127.29.124.135) changed from NON_VOTER to VOTER. New config: { opid_index: 224 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false } } }
I20240501 14:04:45.609457 3667 raft_consensus.cc:348] T 6a5325b7aa824831a5fa056fb4439110 P 4e6923f9753a4450ac4343c34c427d1e [term 2 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 178 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } } peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: true } }
I20240501 14:04:45.611006 3667 raft_consensus.cc:729] T 6a5325b7aa824831a5fa056fb4439110 P 4e6923f9753a4450ac4343c34c427d1e [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 4e6923f9753a4450ac4343c34c427d1e, State: Initialized, Role: LEARNER
I20240501 14:04:45.611233 3667 consensus_queue.cc:260] T 6a5325b7aa824831a5fa056fb4439110 P 4e6923f9753a4450ac4343c34c427d1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 186, Last appended: 2.187, Last appended by leader: 187, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 178 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } } peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: true } }
I20240501 14:04:45.605821 1050 catalog_manager.cc:5484] T 0f11914838ca4139a1c1813fc5334782 P 5d45680f41c84ac39bd97c6f00ac2b06 reported cstate change: config changed from index 196 to 224, 752bc4569da647679d5df51641961840 (127.29.124.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" committed_config { opid_index: 224 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:04:45.634204 3667 ts_tablet_manager.cc:1398] T 6a5325b7aa824831a5fa056fb4439110 P 4e6923f9753a4450ac4343c34c427d1e: Time spent starting tablet: real 0.025s user 0.006s sys 0.000s
I20240501 14:04:45.635986 1343 tablet_copy_service.cc:342] P 05f4ef3d126d43e9be32731e708ff76c: Request end of tablet copy session 4e6923f9753a4450ac4343c34c427d1e-6a5325b7aa824831a5fa056fb4439110 received from {username='slave'} at 127.29.124.133:43219
I20240501 14:04:45.636168 1343 tablet_copy_service.cc:433] P 05f4ef3d126d43e9be32731e708ff76c: ending tablet copy session 4e6923f9753a4450ac4343c34c427d1e-6a5325b7aa824831a5fa056fb4439110 on tablet 6a5325b7aa824831a5fa056fb4439110 with peer 4e6923f9753a4450ac4343c34c427d1e
I20240501 14:04:45.795483 1192 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0f11914838ca4139a1c1813fc5334782"
dest_uuid: "5d45680f41c84ac39bd97c6f00ac2b06"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:49146
I20240501 14:04:45.795737 1192 raft_consensus.cc:595] T 0f11914838ca4139a1c1813fc5334782 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 LEADER]: Received request to transfer leadership
W20240501 14:04:45.826862 1151 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55074: Service unavailable: leader transfer in progress
W20240501 14:04:45.834944 1151 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55074: Service unavailable: leader transfer in progress
W20240501 14:04:45.843932 1151 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55074: Service unavailable: leader transfer in progress
W20240501 14:04:45.857026 1151 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55074: Service unavailable: leader transfer in progress
I20240501 14:04:45.870992 2621 raft_consensus.cc:1203] T 6a5325b7aa824831a5fa056fb4439110 P 4e6923f9753a4450ac4343c34c427d1e [term 2 LEARNER]: Deduplicated request from leader. Original: 2.177->[2.178-2.201] Dedup: 2.187->[2.188-2.201]
W20240501 14:04:45.876950 1151 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55074: Service unavailable: leader transfer in progress
W20240501 14:04:45.885736 1152 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55074: Service unavailable: leader transfer in progress
W20240501 14:04:45.901769 1153 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55074: Service unavailable: leader transfer in progress
W20240501 14:04:45.917870 1153 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55074: Service unavailable: leader transfer in progress
W20240501 14:04:45.939622 1153 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55074: Service unavailable: leader transfer in progress
W20240501 14:04:45.952790 1153 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55074: Service unavailable: leader transfer in progress
I20240501 14:04:45.958752 3542 raft_consensus.cc:1050] T 6a5325b7aa824831a5fa056fb4439110 P 05f4ef3d126d43e9be32731e708ff76c: attempting to promote NON_VOTER 4e6923f9753a4450ac4343c34c427d1e to VOTER
I20240501 14:04:45.959434 3542 consensus_queue.cc:237] T 6a5325b7aa824831a5fa056fb4439110 P 05f4ef3d126d43e9be32731e708ff76c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 202, Committed index: 202, Last appended: 2.202, Last appended by leader: 103, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 203 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } } peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: false } }
I20240501 14:04:45.962507 2488 raft_consensus.cc:1261] T 6a5325b7aa824831a5fa056fb4439110 P 953c332688d14831b3bc7cfd895e06e9 [term 2 FOLLOWER]: Refusing update from remote peer 05f4ef3d126d43e9be32731e708ff76c: Log matching property violated. Preceding OpId in replica: term: 2 index: 202. Preceding OpId from leader: term: 2 index: 203. (index mismatch)
I20240501 14:04:45.963420 3638 consensus_queue.cc:1035] T 6a5325b7aa824831a5fa056fb4439110 P 05f4ef3d126d43e9be32731e708ff76c [LEADER]: Connected to new peer: Peer: permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 203, Last known committed idx: 202, Time since last communication: 0.000s
I20240501 14:04:45.966676 2620 raft_consensus.cc:1261] T 6a5325b7aa824831a5fa056fb4439110 P 4e6923f9753a4450ac4343c34c427d1e [term 2 LEARNER]: Refusing update from remote peer 05f4ef3d126d43e9be32731e708ff76c: Log matching property violated. Preceding OpId in replica: term: 2 index: 202. Preceding OpId from leader: term: 2 index: 203. (index mismatch)
I20240501 14:04:45.967324 3638 consensus_queue.cc:1035] T 6a5325b7aa824831a5fa056fb4439110 P 05f4ef3d126d43e9be32731e708ff76c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 203, Last known committed idx: 201, Time since last communication: 0.000s
W20240501 14:04:45.970913 1153 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55074: Service unavailable: leader transfer in progress
I20240501 14:04:45.971503 3638 raft_consensus.cc:2932] T 6a5325b7aa824831a5fa056fb4439110 P 05f4ef3d126d43e9be32731e708ff76c [term 2 LEADER]: Committing config change with OpId 2.203: config changed from index 178 to 203, 4e6923f9753a4450ac4343c34c427d1e (127.29.124.133) changed from NON_VOTER to VOTER. New config: { opid_index: 203 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } } peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: false } } }
I20240501 14:04:45.973140 2486 raft_consensus.cc:2932] T 6a5325b7aa824831a5fa056fb4439110 P 953c332688d14831b3bc7cfd895e06e9 [term 2 FOLLOWER]: Committing config change with OpId 2.203: config changed from index 178 to 203, 4e6923f9753a4450ac4343c34c427d1e (127.29.124.133) changed from NON_VOTER to VOTER. New config: { opid_index: 203 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } } peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: false } } }
I20240501 14:04:45.977357 1052 catalog_manager.cc:5484] T 6a5325b7aa824831a5fa056fb4439110 P 05f4ef3d126d43e9be32731e708ff76c reported cstate change: config changed from index 178 to 203, 4e6923f9753a4450ac4343c34c427d1e (127.29.124.133) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "05f4ef3d126d43e9be32731e708ff76c" committed_config { opid_index: 203 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:04:45.983431 2621 raft_consensus.cc:2932] T 6a5325b7aa824831a5fa056fb4439110 P 4e6923f9753a4450ac4343c34c427d1e [term 2 FOLLOWER]: Committing config change with OpId 2.203: config changed from index 178 to 203, 4e6923f9753a4450ac4343c34c427d1e (127.29.124.133) changed from NON_VOTER to VOTER. New config: { opid_index: 203 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } } peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: false } } }
W20240501 14:04:45.993760 1151 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55074: Service unavailable: leader transfer in progress
W20240501 14:04:46.010979 1151 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55074: Service unavailable: leader transfer in progress
W20240501 14:04:46.030531 1151 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55074: Service unavailable: leader transfer in progress
W20240501 14:04:46.051244 1151 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55074: Service unavailable: leader transfer in progress
W20240501 14:04:46.069399 1151 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55074: Service unavailable: leader transfer in progress
W20240501 14:04:46.087196 1151 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55074: Service unavailable: leader transfer in progress
W20240501 14:04:46.111410 1151 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55074: Service unavailable: leader transfer in progress
W20240501 14:04:46.134091 1151 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55074: Service unavailable: leader transfer in progress
W20240501 14:04:46.158258 1151 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55074: Service unavailable: leader transfer in progress
W20240501 14:04:46.186170 1151 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55074: Service unavailable: leader transfer in progress
W20240501 14:04:46.228015 1151 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55074: Service unavailable: leader transfer in progress
W20240501 14:04:46.254510 1151 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55074: Service unavailable: leader transfer in progress
I20240501 14:04:46.280962 1323 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6a5325b7aa824831a5fa056fb4439110"
dest_uuid: "05f4ef3d126d43e9be32731e708ff76c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52272
I20240501 14:04:46.281186 1323 raft_consensus.cc:595] T 6a5325b7aa824831a5fa056fb4439110 P 05f4ef3d126d43e9be32731e708ff76c [term 2 LEADER]: Received request to transfer leadership
W20240501 14:04:46.283497 1151 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55074: Service unavailable: leader transfer in progress
I20240501 14:04:46.286568 3599 raft_consensus.cc:979] T 0f11914838ca4139a1c1813fc5334782 P 5d45680f41c84ac39bd97c6f00ac2b06: : Instructing follower 4e6923f9753a4450ac4343c34c427d1e to start an election
I20240501 14:04:46.286711 3599 raft_consensus.cc:1067] T 0f11914838ca4139a1c1813fc5334782 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 LEADER]: Signalling peer 4e6923f9753a4450ac4343c34c427d1e to start an election
I20240501 14:04:46.287771 2621 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0f11914838ca4139a1c1813fc5334782"
dest_uuid: "4e6923f9753a4450ac4343c34c427d1e"
from {username='slave'} at 127.29.124.129:58315
I20240501 14:04:46.288081 2621 raft_consensus.cc:482] T 0f11914838ca4139a1c1813fc5334782 P 4e6923f9753a4450ac4343c34c427d1e [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:04:46.288209 2621 raft_consensus.cc:3037] T 0f11914838ca4139a1c1813fc5334782 P 4e6923f9753a4450ac4343c34c427d1e [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:04:46.291040 2621 raft_consensus.cc:504] T 0f11914838ca4139a1c1813fc5334782 P 4e6923f9753a4450ac4343c34c427d1e [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 224 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false } }
I20240501 14:04:46.294811 2885 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f11914838ca4139a1c1813fc5334782" candidate_uuid: "4e6923f9753a4450ac4343c34c427d1e" candidate_term: 2 candidate_status { last_received { term: 1 index: 230 } } ignore_live_leader: true dest_uuid: "752bc4569da647679d5df51641961840"
I20240501 14:04:46.294972 1192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f11914838ca4139a1c1813fc5334782" candidate_uuid: "4e6923f9753a4450ac4343c34c427d1e" candidate_term: 2 candidate_status { last_received { term: 1 index: 230 } } ignore_live_leader: true dest_uuid: "5d45680f41c84ac39bd97c6f00ac2b06"
I20240501 14:04:46.295028 2885 raft_consensus.cc:3037] T 0f11914838ca4139a1c1813fc5334782 P 752bc4569da647679d5df51641961840 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:04:46.295174 1192 raft_consensus.cc:3032] T 0f11914838ca4139a1c1813fc5334782 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:04:46.295253 1192 raft_consensus.cc:729] T 0f11914838ca4139a1c1813fc5334782 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5d45680f41c84ac39bd97c6f00ac2b06, State: Running, Role: LEADER
I20240501 14:04:46.295545 1192 consensus_queue.cc:260] T 0f11914838ca4139a1c1813fc5334782 P 5d45680f41c84ac39bd97c6f00ac2b06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 230, Committed index: 230, Last appended: 1.230, Last appended by leader: 230, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 224 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false } }
I20240501 14:04:46.295857 1192 raft_consensus.cc:3037] T 0f11914838ca4139a1c1813fc5334782 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:04:46.298301 2885 raft_consensus.cc:2454] T 0f11914838ca4139a1c1813fc5334782 P 752bc4569da647679d5df51641961840 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e6923f9753a4450ac4343c34c427d1e in term 2.
I20240501 14:04:46.298506 2621 leader_election.cc:288] T 0f11914838ca4139a1c1813fc5334782 P 4e6923f9753a4450ac4343c34c427d1e [CANDIDATE]: Term 2 election: Requested vote from peers 5d45680f41c84ac39bd97c6f00ac2b06 (127.29.124.129:40925), 752bc4569da647679d5df51641961840 (127.29.124.135:42923)
I20240501 14:04:46.299021 2555 leader_election.cc:302] T 0f11914838ca4139a1c1813fc5334782 P 4e6923f9753a4450ac4343c34c427d1e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e6923f9753a4450ac4343c34c427d1e, 752bc4569da647679d5df51641961840; no voters:
I20240501 14:04:46.299378 3702 raft_consensus.cc:2781] T 0f11914838ca4139a1c1813fc5334782 P 4e6923f9753a4450ac4343c34c427d1e [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:04:46.304471 1192 raft_consensus.cc:2454] T 0f11914838ca4139a1c1813fc5334782 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e6923f9753a4450ac4343c34c427d1e in term 2.
I20240501 14:04:46.304706 3702 raft_consensus.cc:686] T 0f11914838ca4139a1c1813fc5334782 P 4e6923f9753a4450ac4343c34c427d1e [term 2 LEADER]: Becoming Leader. State: Replica: 4e6923f9753a4450ac4343c34c427d1e, State: Running, Role: LEADER
I20240501 14:04:46.305413 3702 consensus_queue.cc:237] T 0f11914838ca4139a1c1813fc5334782 P 4e6923f9753a4450ac4343c34c427d1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 230, Committed index: 230, Last appended: 1.230, Last appended by leader: 230, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 224 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false } }
I20240501 14:04:46.309201 1052 catalog_manager.cc:5484] T 0f11914838ca4139a1c1813fc5334782 P 4e6923f9753a4450ac4343c34c427d1e reported cstate change: term changed from 1 to 2, leader changed from 5d45680f41c84ac39bd97c6f00ac2b06 (127.29.124.129) to 4e6923f9753a4450ac4343c34c427d1e (127.29.124.133). New cstate: current_term: 2 leader_uuid: "4e6923f9753a4450ac4343c34c427d1e" committed_config { opid_index: 224 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240501 14:04:46.319849 1151 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:55074: Illegal state: replica 5d45680f41c84ac39bd97c6f00ac2b06 is not leader of this config: current role FOLLOWER
W20240501 14:04:46.367350 1281 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58188: Service unavailable: leader transfer in progress
W20240501 14:04:46.375770 1281 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58188: Service unavailable: leader transfer in progress
W20240501 14:04:46.384442 1282 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58188: Service unavailable: leader transfer in progress
W20240501 14:04:46.392339 1282 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58188: Service unavailable: leader transfer in progress
W20240501 14:04:46.403210 1282 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58188: Service unavailable: leader transfer in progress
W20240501 14:04:46.410832 1282 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58188: Service unavailable: leader transfer in progress
W20240501 14:04:46.420071 1282 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58188: Service unavailable: leader transfer in progress
W20240501 14:04:46.432998 1282 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58188: Service unavailable: leader transfer in progress
I20240501 14:04:46.438778 2885 raft_consensus.cc:1261] T 0f11914838ca4139a1c1813fc5334782 P 752bc4569da647679d5df51641961840 [term 2 FOLLOWER]: Refusing update from remote peer 4e6923f9753a4450ac4343c34c427d1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 230. Preceding OpId from leader: term: 2 index: 232. (index mismatch)
I20240501 14:04:46.438941 1192 raft_consensus.cc:1261] T 0f11914838ca4139a1c1813fc5334782 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 2 FOLLOWER]: Refusing update from remote peer 4e6923f9753a4450ac4343c34c427d1e: Log matching property violated. Preceding OpId in replica: term: 1 index: 230. Preceding OpId from leader: term: 2 index: 232. (index mismatch)
I20240501 14:04:46.440315 3738 consensus_queue.cc:1035] T 0f11914838ca4139a1c1813fc5334782 P 4e6923f9753a4450ac4343c34c427d1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 231, Last known committed idx: 230, Time since last communication: 0.000s
I20240501 14:04:46.440852 3702 consensus_queue.cc:1035] T 0f11914838ca4139a1c1813fc5334782 P 4e6923f9753a4450ac4343c34c427d1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 231, Last known committed idx: 230, Time since last communication: 0.000s
I20240501 14:04:46.450330 3547 mvcc.cc:205] Tried to move back new op lower bound from 7022888085244547072 to 7022888084712181760. Current Snapshot: MvccSnapshot[applied={T|T < 7022888085244547072 or (T in {7022888085244547072})}]
W20240501 14:04:46.458810 1282 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58188: Service unavailable: leader transfer in progress
W20240501 14:04:46.473380 1282 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58188: Service unavailable: leader transfer in progress
W20240501 14:04:46.491609 1282 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58188: Service unavailable: leader transfer in progress
I20240501 14:04:46.493402 2620 consensus_queue.cc:237] T 0f11914838ca4139a1c1813fc5334782 P 4e6923f9753a4450ac4343c34c427d1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 233, Committed index: 233, Last appended: 2.233, Last appended by leader: 230, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false } }
I20240501 14:04:46.496809 2885 raft_consensus.cc:1261] T 0f11914838ca4139a1c1813fc5334782 P 752bc4569da647679d5df51641961840 [term 2 FOLLOWER]: Refusing update from remote peer 4e6923f9753a4450ac4343c34c427d1e: Log matching property violated. Preceding OpId in replica: term: 2 index: 233. Preceding OpId from leader: term: 2 index: 234. (index mismatch)
I20240501 14:04:46.499552 3738 consensus_queue.cc:1035] T 0f11914838ca4139a1c1813fc5334782 P 4e6923f9753a4450ac4343c34c427d1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 234, Last known committed idx: 233, Time since last communication: 0.000s
I20240501 14:04:46.502862 3738 raft_consensus.cc:2932] T 0f11914838ca4139a1c1813fc5334782 P 4e6923f9753a4450ac4343c34c427d1e [term 2 LEADER]: Committing config change with OpId 2.234: config changed from index 224 to 234, VOTER 5d45680f41c84ac39bd97c6f00ac2b06 (127.29.124.129) evicted. New config: { opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false } } }
I20240501 14:04:46.506726 2885 raft_consensus.cc:2932] T 0f11914838ca4139a1c1813fc5334782 P 752bc4569da647679d5df51641961840 [term 2 FOLLOWER]: Committing config change with OpId 2.234: config changed from index 224 to 234, VOTER 5d45680f41c84ac39bd97c6f00ac2b06 (127.29.124.129) evicted. New config: { opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: false } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false } } }
W20240501 14:04:46.507380 1282 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58188: Service unavailable: leader transfer in progress
I20240501 14:04:46.516348 1036 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0f11914838ca4139a1c1813fc5334782 with cas_config_opid_index 224: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:04:46.518955 1045 catalog_manager.cc:5484] T 0f11914838ca4139a1c1813fc5334782 P 4e6923f9753a4450ac4343c34c427d1e reported cstate change: config changed from index 224 to 234, VOTER 5d45680f41c84ac39bd97c6f00ac2b06 (127.29.124.129) evicted. New cstate: current_term: 2 leader_uuid: "4e6923f9753a4450ac4343c34c427d1e" committed_config { opid_index: 234 OBSOLETE_local: false peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:04:46.523044 1172 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f11914838ca4139a1c1813fc5334782 with delete_type TABLET_DATA_TOMBSTONED (TS 5d45680f41c84ac39bd97c6f00ac2b06 not found in new config with opid_index 234) from {username='slave'} at 127.0.0.1:51144
I20240501 14:04:46.523838 3752 tablet_replica.cc:330] T 0f11914838ca4139a1c1813fc5334782 P 5d45680f41c84ac39bd97c6f00ac2b06: stopping tablet replica
I20240501 14:04:46.524132 3752 raft_consensus.cc:2229] T 0f11914838ca4139a1c1813fc5334782 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 2 FOLLOWER]: Raft consensus shutting down.
W20240501 14:04:46.524271 1282 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58188: Service unavailable: leader transfer in progress
I20240501 14:04:46.524466 3752 raft_consensus.cc:2258] T 0f11914838ca4139a1c1813fc5334782 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:04:46.525907 3752 ts_tablet_manager.cc:1875] T 0f11914838ca4139a1c1813fc5334782 P 5d45680f41c84ac39bd97c6f00ac2b06: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:04:46.533483 3752 ts_tablet_manager.cc:1888] T 0f11914838ca4139a1c1813fc5334782 P 5d45680f41c84ac39bd97c6f00ac2b06: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.233
I20240501 14:04:46.534163 3752 log.cc:1196] T 0f11914838ca4139a1c1813fc5334782 P 5d45680f41c84ac39bd97c6f00ac2b06: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/wal/wals/0f11914838ca4139a1c1813fc5334782
I20240501 14:04:46.535111 1036 catalog_manager.cc:4830] TS 5d45680f41c84ac39bd97c6f00ac2b06 (127.29.124.129:40925): tablet 0f11914838ca4139a1c1813fc5334782 (table rebalance_test_table_3 [id=16c6cbe8fef84c88b36c2d1809ebe48b]) successfully deleted
W20240501 14:04:46.542848 1283 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58188: Service unavailable: leader transfer in progress
W20240501 14:04:46.562587 1283 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58188: Service unavailable: leader transfer in progress
W20240501 14:04:46.585225 1282 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58188: Service unavailable: leader transfer in progress
W20240501 14:04:46.605631 1282 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58188: Service unavailable: leader transfer in progress
W20240501 14:04:46.635325 1282 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58188: Service unavailable: leader transfer in progress
W20240501 14:04:46.658301 1282 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58188: Service unavailable: leader transfer in progress
W20240501 14:04:46.682376 1282 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58188: Service unavailable: leader transfer in progress
I20240501 14:04:46.696830 1323 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6a5325b7aa824831a5fa056fb4439110"
dest_uuid: "05f4ef3d126d43e9be32731e708ff76c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52298
I20240501 14:04:46.697072 1323 raft_consensus.cc:595] T 6a5325b7aa824831a5fa056fb4439110 P 05f4ef3d126d43e9be32731e708ff76c [term 2 LEADER]: Received request to transfer leadership
W20240501 14:04:46.709787 1282 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58188: Service unavailable: leader transfer in progress
W20240501 14:04:46.736304 1282 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58188: Service unavailable: leader transfer in progress
W20240501 14:04:46.761868 1282 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58188: Service unavailable: leader transfer in progress
I20240501 14:04:46.766090 3631 raft_consensus.cc:979] T 6a5325b7aa824831a5fa056fb4439110 P 05f4ef3d126d43e9be32731e708ff76c: : Instructing follower 4e6923f9753a4450ac4343c34c427d1e to start an election
I20240501 14:04:46.766233 3631 raft_consensus.cc:1067] T 6a5325b7aa824831a5fa056fb4439110 P 05f4ef3d126d43e9be32731e708ff76c [term 2 LEADER]: Signalling peer 4e6923f9753a4450ac4343c34c427d1e to start an election
I20240501 14:04:46.767357 2621 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6a5325b7aa824831a5fa056fb4439110"
dest_uuid: "4e6923f9753a4450ac4343c34c427d1e"
from {username='slave'} at 127.29.124.130:44311
I20240501 14:04:46.767550 2621 raft_consensus.cc:482] T 6a5325b7aa824831a5fa056fb4439110 P 4e6923f9753a4450ac4343c34c427d1e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:04:46.767663 2621 raft_consensus.cc:3037] T 6a5325b7aa824831a5fa056fb4439110 P 4e6923f9753a4450ac4343c34c427d1e [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:04:46.770574 2621 raft_consensus.cc:504] T 6a5325b7aa824831a5fa056fb4439110 P 4e6923f9753a4450ac4343c34c427d1e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 203 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } } peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: false } }
I20240501 14:04:46.771440 2621 leader_election.cc:288] T 6a5325b7aa824831a5fa056fb4439110 P 4e6923f9753a4450ac4343c34c427d1e [CANDIDATE]: Term 3 election: Requested vote from peers 05f4ef3d126d43e9be32731e708ff76c (127.29.124.130:41511), 953c332688d14831b3bc7cfd895e06e9 (127.29.124.132:34305)
I20240501 14:04:46.772845 1323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a5325b7aa824831a5fa056fb4439110" candidate_uuid: "4e6923f9753a4450ac4343c34c427d1e" candidate_term: 3 candidate_status { last_received { term: 2 index: 214 } } ignore_live_leader: true dest_uuid: "05f4ef3d126d43e9be32731e708ff76c"
I20240501 14:04:46.773051 1323 raft_consensus.cc:3032] T 6a5325b7aa824831a5fa056fb4439110 P 05f4ef3d126d43e9be32731e708ff76c [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:04:46.773131 1323 raft_consensus.cc:729] T 6a5325b7aa824831a5fa056fb4439110 P 05f4ef3d126d43e9be32731e708ff76c [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 05f4ef3d126d43e9be32731e708ff76c, State: Running, Role: LEADER
I20240501 14:04:46.773401 1323 consensus_queue.cc:260] T 6a5325b7aa824831a5fa056fb4439110 P 05f4ef3d126d43e9be32731e708ff76c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 214, Committed index: 214, Last appended: 2.214, Last appended by leader: 214, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 203 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } } peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: false } }
I20240501 14:04:46.773727 1323 raft_consensus.cc:3037] T 6a5325b7aa824831a5fa056fb4439110 P 05f4ef3d126d43e9be32731e708ff76c [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:04:46.776298 1323 raft_consensus.cc:2454] T 6a5325b7aa824831a5fa056fb4439110 P 05f4ef3d126d43e9be32731e708ff76c [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e6923f9753a4450ac4343c34c427d1e in term 3.
W20240501 14:04:46.790607 1282 tablet_service.cc:695] failed op from {username='slave'} at 127.0.0.1:58188: Illegal state: replica 05f4ef3d126d43e9be32731e708ff76c is not leader of this config: current role FOLLOWER
I20240501 14:04:46.806161 2555 leader_election.cc:302] T 6a5325b7aa824831a5fa056fb4439110 P 4e6923f9753a4450ac4343c34c427d1e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 05f4ef3d126d43e9be32731e708ff76c, 4e6923f9753a4450ac4343c34c427d1e; no voters:
I20240501 14:04:46.806740 3749 raft_consensus.cc:2781] T 6a5325b7aa824831a5fa056fb4439110 P 4e6923f9753a4450ac4343c34c427d1e [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:04:46.806890 3749 raft_consensus.cc:686] T 6a5325b7aa824831a5fa056fb4439110 P 4e6923f9753a4450ac4343c34c427d1e [term 3 LEADER]: Becoming Leader. State: Replica: 4e6923f9753a4450ac4343c34c427d1e, State: Running, Role: LEADER
I20240501 14:04:46.807201 3749 consensus_queue.cc:237] T 6a5325b7aa824831a5fa056fb4439110 P 4e6923f9753a4450ac4343c34c427d1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 214, Committed index: 214, Last appended: 2.214, Last appended by leader: 214, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 203 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } } peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: false } }
I20240501 14:04:46.810849 1052 catalog_manager.cc:5484] T 6a5325b7aa824831a5fa056fb4439110 P 4e6923f9753a4450ac4343c34c427d1e reported cstate change: term changed from 2 to 3, leader changed from 05f4ef3d126d43e9be32731e708ff76c (127.29.124.130) to 4e6923f9753a4450ac4343c34c427d1e (127.29.124.133). New cstate: current_term: 3 leader_uuid: "4e6923f9753a4450ac4343c34c427d1e" committed_config { opid_index: 203 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:04:46.817530 2486 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a5325b7aa824831a5fa056fb4439110" candidate_uuid: "4e6923f9753a4450ac4343c34c427d1e" candidate_term: 3 candidate_status { last_received { term: 2 index: 214 } } ignore_live_leader: true dest_uuid: "953c332688d14831b3bc7cfd895e06e9"
I20240501 14:04:46.817713 2486 raft_consensus.cc:3037] T 6a5325b7aa824831a5fa056fb4439110 P 953c332688d14831b3bc7cfd895e06e9 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:04:46.820894 2486 raft_consensus.cc:2454] T 6a5325b7aa824831a5fa056fb4439110 P 953c332688d14831b3bc7cfd895e06e9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e6923f9753a4450ac4343c34c427d1e in term 3.
I20240501 14:04:46.919355 2488 raft_consensus.cc:1261] T 6a5325b7aa824831a5fa056fb4439110 P 953c332688d14831b3bc7cfd895e06e9 [term 3 FOLLOWER]: Refusing update from remote peer 4e6923f9753a4450ac4343c34c427d1e: Log matching property violated. Preceding OpId in replica: term: 2 index: 214. Preceding OpId from leader: term: 3 index: 216. (index mismatch)
I20240501 14:04:46.919862 3749 consensus_queue.cc:1035] T 6a5325b7aa824831a5fa056fb4439110 P 4e6923f9753a4450ac4343c34c427d1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 215, Last known committed idx: 214, Time since last communication: 0.000s
I20240501 14:04:46.921370 1323 raft_consensus.cc:1261] T 6a5325b7aa824831a5fa056fb4439110 P 05f4ef3d126d43e9be32731e708ff76c [term 3 FOLLOWER]: Refusing update from remote peer 4e6923f9753a4450ac4343c34c427d1e: Log matching property violated. Preceding OpId in replica: term: 2 index: 214. Preceding OpId from leader: term: 3 index: 216. (index mismatch)
I20240501 14:04:46.922060 3749 consensus_queue.cc:1035] T 6a5325b7aa824831a5fa056fb4439110 P 4e6923f9753a4450ac4343c34c427d1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 215, Last known committed idx: 214, Time since last communication: 0.000s
I20240501 14:04:46.945555 2620 consensus_queue.cc:237] T 6a5325b7aa824831a5fa056fb4439110 P 4e6923f9753a4450ac4343c34c427d1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 216, Committed index: 216, Last appended: 3.216, Last appended by leader: 214, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } } peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: false } }
I20240501 14:04:46.955392 2487 raft_consensus.cc:1261] T 6a5325b7aa824831a5fa056fb4439110 P 953c332688d14831b3bc7cfd895e06e9 [term 3 FOLLOWER]: Refusing update from remote peer 4e6923f9753a4450ac4343c34c427d1e: Log matching property violated. Preceding OpId in replica: term: 3 index: 216. Preceding OpId from leader: term: 3 index: 218. (index mismatch)
I20240501 14:04:46.958918 3738 consensus_queue.cc:1035] T 6a5325b7aa824831a5fa056fb4439110 P 4e6923f9753a4450ac4343c34c427d1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 217, Last known committed idx: 216, Time since last communication: 0.000s
I20240501 14:04:46.965282 3702 raft_consensus.cc:2932] T 6a5325b7aa824831a5fa056fb4439110 P 4e6923f9753a4450ac4343c34c427d1e [term 3 LEADER]: Committing config change with OpId 3.217: config changed from index 203 to 217, VOTER 05f4ef3d126d43e9be32731e708ff76c (127.29.124.130) evicted. New config: { opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } } peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: false } } }
I20240501 14:04:46.966459 2486 raft_consensus.cc:2932] T 6a5325b7aa824831a5fa056fb4439110 P 953c332688d14831b3bc7cfd895e06e9 [term 3 FOLLOWER]: Committing config change with OpId 3.217: config changed from index 203 to 217, VOTER 05f4ef3d126d43e9be32731e708ff76c (127.29.124.130) evicted. New config: { opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } } peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: false } } }
I20240501 14:04:46.969221 1036 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6a5325b7aa824831a5fa056fb4439110 with cas_config_opid_index 203: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:04:46.981487 1052 catalog_manager.cc:5484] T 6a5325b7aa824831a5fa056fb4439110 P 4e6923f9753a4450ac4343c34c427d1e reported cstate change: config changed from index 203 to 217, VOTER 05f4ef3d126d43e9be32731e708ff76c (127.29.124.130) evicted. New cstate: current_term: 3 leader_uuid: "4e6923f9753a4450ac4343c34c427d1e" committed_config { opid_index: 217 OBSOLETE_local: false peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:04:46.987876 1302 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a5325b7aa824831a5fa056fb4439110 with delete_type TABLET_DATA_TOMBSTONED (TS 05f4ef3d126d43e9be32731e708ff76c not found in new config with opid_index 217) from {username='slave'} at 127.0.0.1:35916
I20240501 14:04:46.993665 3786 tablet_replica.cc:330] T 6a5325b7aa824831a5fa056fb4439110 P 05f4ef3d126d43e9be32731e708ff76c: stopping tablet replica
I20240501 14:04:46.993976 3786 raft_consensus.cc:2229] T 6a5325b7aa824831a5fa056fb4439110 P 05f4ef3d126d43e9be32731e708ff76c [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 14:04:46.994341 3786 raft_consensus.cc:2258] T 6a5325b7aa824831a5fa056fb4439110 P 05f4ef3d126d43e9be32731e708ff76c [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 14:04:47.050032 3786 ts_tablet_manager.cc:1875] T 6a5325b7aa824831a5fa056fb4439110 P 05f4ef3d126d43e9be32731e708ff76c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:04:47.083741 3786 ts_tablet_manager.cc:1888] T 6a5325b7aa824831a5fa056fb4439110 P 05f4ef3d126d43e9be32731e708ff76c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.216
I20240501 14:04:47.084579 3786 log.cc:1196] T 6a5325b7aa824831a5fa056fb4439110 P 05f4ef3d126d43e9be32731e708ff76c: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/wal/wals/6a5325b7aa824831a5fa056fb4439110
I20240501 14:04:47.085740 1038 catalog_manager.cc:4830] TS 05f4ef3d126d43e9be32731e708ff76c (127.29.124.130:41511): tablet 6a5325b7aa824831a5fa056fb4439110 (table rebalance_test_table_1 [id=423117c09096433e93fec4726745a428]) successfully deleted
I20240501 14:04:47.967315 1302 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:48.050297 2733 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:04:48.101164 2469 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:04:48.174834 2864 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:04:48.188576 2602 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:04:48.219270 1172 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:48.389417 1432 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:04:48.995647 1302 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:49.118417 2733 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:04:49.151122 1172 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:49.206935 2469 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:04:49.224541 2602 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:04:49.248744 1432 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:04:49.343518 2864 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:04:51.147429 1302 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:51.224238 2602 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:04:51.243337 2733 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:04:51.275768 1432 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:04:51.294996 1172 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:51.299546 2864 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:04:51.328971 2469 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:04:52.155926 1302 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:52.205376 2864 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:04:52.218376 2733 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:04:52.219239 2602 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:04:52.224234 1432 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:04:52.298081 1172 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:52.363174 2469 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:04:52.828830 1302 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:52.865398 1172 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:52.867990 2733 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:04:52.895360 2602 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:04:52.896071 2864 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:04:52.902760 2469 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:04:52.926723 1432 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
3e721134e18641eca52cbebd71ec0520 | 127.29.124.190:46779 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.29.124.148:36129 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
05f4ef3d126d43e9be32731e708ff76c | 127.29.124.130:41511 | HEALTHY | <none> | 5 | 0
32064197afde4b79bf388b080c365d88 | 127.29.124.134:37211 | HEALTHY | <none> | 0 | 0
4e6923f9753a4450ac4343c34c427d1e | 127.29.124.133:34675 | HEALTHY | <none> | 2 | 0
5d45680f41c84ac39bd97c6f00ac2b06 | 127.29.124.129:40925 | HEALTHY | <none> | 4 | 0
752bc4569da647679d5df51641961840 | 127.29.124.135:42923 | HEALTHY | <none> | 2 | 0
953c332688d14831b3bc7cfd895e06e9 | 127.29.124.132:34305 | HEALTHY | <none> | 0 | 0
eb004c2ac9d244f89b61448584d4115b | 127.29.124.131:40225 | HEALTHY | <none> | 3 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.29.124.129 | experimental | 127.29.124.129:40925
local_ip_for_outbound_sockets | 127.29.124.130 | experimental | 127.29.124.130:41511
local_ip_for_outbound_sockets | 127.29.124.131 | experimental | 127.29.124.131:40225
local_ip_for_outbound_sockets | 127.29.124.132 | experimental | 127.29.124.132:34305
local_ip_for_outbound_sockets | 127.29.124.133 | experimental | 127.29.124.133:34675
local_ip_for_outbound_sockets | 127.29.124.134 | experimental | 127.29.124.134:37211
local_ip_for_outbound_sockets | 127.29.124.135 | experimental | 127.29.124.135:42923
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.29.124.129:40925
server_dump_info_path | /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.29.124.130:41511
server_dump_info_path | /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.29.124.131:40225
server_dump_info_path | /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.29.124.132:34305
server_dump_info_path | /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.29.124.133:34675
server_dump_info_path | /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.29.124.134:37211
server_dump_info_path | /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.29.124.135:42923
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.29.124.148:36129 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 4
First Quartile | 4
Median | 5
Third Quartile | 5
Maximum | 5
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 32
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240501 14:04:52.994864 30194 log_verifier.cc:126] Checking tablet 0f11914838ca4139a1c1813fc5334782
I20240501 14:04:53.134089 30194 log_verifier.cc:177] Verified matching terms for 366 ops in tablet 0f11914838ca4139a1c1813fc5334782
I20240501 14:04:53.134914 30194 log_verifier.cc:126] Checking tablet 0f837cac32ad4aaa970124639138c2db
I20240501 14:04:53.293848 30194 log_verifier.cc:177] Verified matching terms for 383 ops in tablet 0f837cac32ad4aaa970124639138c2db
I20240501 14:04:53.294399 30194 log_verifier.cc:126] Checking tablet 1074c96bc2564edca376f95bd195a62c
I20240501 14:04:53.432343 30194 log_verifier.cc:177] Verified matching terms for 404 ops in tablet 1074c96bc2564edca376f95bd195a62c
I20240501 14:04:53.432863 30194 log_verifier.cc:126] Checking tablet 14e026c65736401fb3aec7e6462ae001
I20240501 14:04:53.573879 30194 log_verifier.cc:177] Verified matching terms for 359 ops in tablet 14e026c65736401fb3aec7e6462ae001
I20240501 14:04:53.574347 30194 log_verifier.cc:126] Checking tablet 14e0e4808438484688605ff2d6d9258b
I20240501 14:04:53.696192 30194 log_verifier.cc:177] Verified matching terms for 364 ops in tablet 14e0e4808438484688605ff2d6d9258b
I20240501 14:04:53.696632 30194 log_verifier.cc:126] Checking tablet 3a4ca3f4edcd4bf29d0bab40a8856039
I20240501 14:04:53.827219 30194 log_verifier.cc:177] Verified matching terms for 380 ops in tablet 3a4ca3f4edcd4bf29d0bab40a8856039
I20240501 14:04:53.827683 30194 log_verifier.cc:126] Checking tablet 45d428dc499a4270b56a579f5b3cad83
I20240501 14:04:53.962532 30194 log_verifier.cc:177] Verified matching terms for 376 ops in tablet 45d428dc499a4270b56a579f5b3cad83
I20240501 14:04:53.963083 30194 log_verifier.cc:126] Checking tablet 4c22d4541da544f5b812c6e5e8c2ea40
I20240501 14:04:54.091243 30194 log_verifier.cc:177] Verified matching terms for 357 ops in tablet 4c22d4541da544f5b812c6e5e8c2ea40
I20240501 14:04:54.091687 30194 log_verifier.cc:126] Checking tablet 4da90afe36fd411c8bfd661e4881f86c
I20240501 14:04:54.117511 30194 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 4da90afe36fd411c8bfd661e4881f86c
I20240501 14:04:54.117604 30194 log_verifier.cc:126] Checking tablet 53249c71dab94ff4b179309088f4d1cd
I20240501 14:04:54.251549 30194 log_verifier.cc:177] Verified matching terms for 362 ops in tablet 53249c71dab94ff4b179309088f4d1cd
I20240501 14:04:54.251989 30194 log_verifier.cc:126] Checking tablet 6a5325b7aa824831a5fa056fb4439110
I20240501 14:04:54.372035 30194 log_verifier.cc:177] Verified matching terms for 346 ops in tablet 6a5325b7aa824831a5fa056fb4439110
I20240501 14:04:54.372447 30194 log_verifier.cc:126] Checking tablet 70afcb72943b49a7bea962791b38a91d
I20240501 14:04:54.490835 30194 log_verifier.cc:177] Verified matching terms for 345 ops in tablet 70afcb72943b49a7bea962791b38a91d
I20240501 14:04:54.491281 30194 log_verifier.cc:126] Checking tablet 8979fd4d21814622ba687edb88fa7851
I20240501 14:04:54.631094 30194 log_verifier.cc:177] Verified matching terms for 406 ops in tablet 8979fd4d21814622ba687edb88fa7851
I20240501 14:04:54.631585 30194 log_verifier.cc:126] Checking tablet c586d994d2414073b25a8b3fab6a7046
I20240501 14:04:54.767885 30194 log_verifier.cc:177] Verified matching terms for 355 ops in tablet c586d994d2414073b25a8b3fab6a7046
I20240501 14:04:54.768334 30194 log_verifier.cc:126] Checking tablet db2b3f73d20c4790808bbe0f1d5e489c
I20240501 14:04:54.895177 30194 log_verifier.cc:177] Verified matching terms for 361 ops in tablet db2b3f73d20c4790808bbe0f1d5e489c
I20240501 14:04:54.895637 30194 log_verifier.cc:126] Checking tablet ef358ca6b48646638ec351cb06cf704d
I20240501 14:04:55.012001 30194 log_verifier.cc:177] Verified matching terms for 306 ops in tablet ef358ca6b48646638ec351cb06cf704d
I20240501 14:04:55.015460 30194 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUv7aek/build/asan/bin/kudu
/tmp/dist-test-taskUv7aek/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.124.136:0
--local_ip_for_outbound_sockets=127.29.124.136
--webserver_interface=127.29.124.136
--webserver_port=0
--tserver_master_addrs=127.29.124.190:46779
--builtin_ntp_servers=127.29.124.148:36129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 14:04:55.340642 4074 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:04:55.340909 4074 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:04:55.341054 4074 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:04:55.348435 4074 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:04:55.348560 4074 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:04:55.348726 4074 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.124.136
I20240501 14:04:55.357753 4074 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.124.148:36129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.124.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.29.124.136
--webserver_port=0
--tserver_master_addrs=127.29.124.190:46779
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.124.136
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:04:55.358952 4074 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:04:55.360584 4074 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:04:55.371825 4079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:04:55.372365 4080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:04:55.372701 4082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:04:55.373131 4074 server_base.cc:1034] running on GCE node
I20240501 14:04:55.373706 4074 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:04:55.374692 4074 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:04:55.375914 4074 hybrid_clock.cc:648] HybridClock initialized: now 1714572295375866 us; error 39 us; skew 500 ppm
I20240501 14:04:55.376266 4074 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:04:55.382242 4074 webserver.cc:460] Webserver started at http://127.29.124.136:39107/ using document root <none> and password file <none>
I20240501 14:04:55.382957 4074 fs_manager.cc:359] Metadata directory not provided
I20240501 14:04:55.383059 4074 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:04:55.383347 4074 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:04:55.385833 4074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "1be9932ac665476aa1f5ba5b69c73dfa"
format_stamp: "Formatted at 2024-05-01 14:04:55 on dist-test-slave-p83k"
I20240501 14:04:55.386556 4074 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "1be9932ac665476aa1f5ba5b69c73dfa"
format_stamp: "Formatted at 2024-05-01 14:04:55 on dist-test-slave-p83k"
I20240501 14:04:55.391347 4074 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20240501 14:04:55.394677 4088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:04:55.395961 4074 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 14:04:55.396103 4074 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "1be9932ac665476aa1f5ba5b69c73dfa"
format_stamp: "Formatted at 2024-05-01 14:04:55 on dist-test-slave-p83k"
I20240501 14:04:55.396281 4074 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:04:55.416579 4074 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:04:55.417353 4074 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:04:55.417611 4074 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:04:55.418792 4074 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:04:55.420348 4074 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:04:55.420428 4074 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:04:55.420518 4074 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:04:55.420588 4074 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:04:55.447433 4074 rpc_server.cc:307] RPC server started. Bound to: 127.29.124.136:39469
I20240501 14:04:55.447480 4200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.124.136:39469 every 8 connection(s)
I20240501 14:04:55.448674 4074 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 14:04:55.451143 30194 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUv7aek/build/asan/bin/kudu as pid 4074
I20240501 14:04:55.451339 30194 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240501 14:04:55.461079 4201 heartbeater.cc:347] Connected to a master server at 127.29.124.190:46779
I20240501 14:04:55.461400 4201 heartbeater.cc:464] Registering TS with master...
I20240501 14:04:55.462163 4201 heartbeater.cc:511] Master 127.29.124.190:46779 requested a full tablet report, sending...
I20240501 14:04:55.463485 1050 ts_manager.cc:194] Registered new tserver with Master: 1be9932ac665476aa1f5ba5b69c73dfa (127.29.124.136:39469)
I20240501 14:04:55.464306 1050 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.124.136:44437
I20240501 14:04:55.905117 2733 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:04:55.923799 1302 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:55.925863 4136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:04:55.946346 2602 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:04:55.950253 2864 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:04:55.977038 1172 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:56.029348 2469 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:04:56.050186 1432 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:04:56.197477 1302 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:04:56.212268 2733 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:04:56.224004 2602 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:04:56.242053 4136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:04:56.250087 1172 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:56.273288 2469 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:04:56.277617 2864 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:04:56.277845 1432 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:04:56.354485 1192 consensus_queue.cc:237] T c586d994d2414073b25a8b3fab6a7046 P 5d45680f41c84ac39bd97c6f00ac2b06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 355, Committed index: 355, Last appended: 1.355, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 356 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: true } }
I20240501 14:04:56.356860 2487 raft_consensus.cc:1261] T c586d994d2414073b25a8b3fab6a7046 P 953c332688d14831b3bc7cfd895e06e9 [term 1 FOLLOWER]: Refusing update from remote peer 5d45680f41c84ac39bd97c6f00ac2b06: Log matching property violated. Preceding OpId in replica: term: 1 index: 355. Preceding OpId from leader: term: 1 index: 356. (index mismatch)
I20240501 14:04:56.357453 4296 consensus_queue.cc:1035] T c586d994d2414073b25a8b3fab6a7046 P 5d45680f41c84ac39bd97c6f00ac2b06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 356, Last known committed idx: 355, Time since last communication: 0.000s
I20240501 14:04:56.360132 3964 raft_consensus.cc:2932] T c586d994d2414073b25a8b3fab6a7046 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 LEADER]: Committing config change with OpId 1.356: config changed from index 67 to 356, NON_VOTER 1be9932ac665476aa1f5ba5b69c73dfa (127.29.124.136) added. New config: { opid_index: 356 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: true } } }
I20240501 14:04:56.361572 2487 raft_consensus.cc:2932] T c586d994d2414073b25a8b3fab6a7046 P 953c332688d14831b3bc7cfd895e06e9 [term 1 FOLLOWER]: Committing config change with OpId 1.356: config changed from index 67 to 356, NON_VOTER 1be9932ac665476aa1f5ba5b69c73dfa (127.29.124.136) added. New config: { opid_index: 356 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: true } } }
W20240501 14:04:56.362985 1129 consensus_peers.cc:485] T c586d994d2414073b25a8b3fab6a7046 P 5d45680f41c84ac39bd97c6f00ac2b06 -> Peer 1be9932ac665476aa1f5ba5b69c73dfa (127.29.124.136:39469): Couldn't send request to peer 1be9932ac665476aa1f5ba5b69c73dfa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c586d994d2414073b25a8b3fab6a7046. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:04:56.365533 1052 catalog_manager.cc:5484] T c586d994d2414073b25a8b3fab6a7046 P 5d45680f41c84ac39bd97c6f00ac2b06 reported cstate change: config changed from index 67 to 356, NON_VOTER 1be9932ac665476aa1f5ba5b69c73dfa (127.29.124.136) added. New cstate: current_term: 1 leader_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" committed_config { opid_index: 356 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:04:56.376507 1323 consensus_queue.cc:237] T 14e0e4808438484688605ff2d6d9258b P 05f4ef3d126d43e9be32731e708ff76c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 364, Committed index: 364, Last appended: 1.364, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 365 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: true } }
I20240501 14:04:56.379168 1449 raft_consensus.cc:1261] T 14e0e4808438484688605ff2d6d9258b P eb004c2ac9d244f89b61448584d4115b [term 1 FOLLOWER]: Refusing update from remote peer 05f4ef3d126d43e9be32731e708ff76c: Log matching property violated. Preceding OpId in replica: term: 1 index: 364. Preceding OpId from leader: term: 1 index: 365. (index mismatch)
I20240501 14:04:56.379738 3896 consensus_queue.cc:1035] T 14e0e4808438484688605ff2d6d9258b P 05f4ef3d126d43e9be32731e708ff76c [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 365, Last known committed idx: 364, Time since last communication: 0.000s
I20240501 14:04:56.382469 4310 raft_consensus.cc:2932] T 14e0e4808438484688605ff2d6d9258b P 05f4ef3d126d43e9be32731e708ff76c [term 1 LEADER]: Committing config change with OpId 1.365: config changed from index -1 to 365, NON_VOTER 1be9932ac665476aa1f5ba5b69c73dfa (127.29.124.136) added. New config: { opid_index: 365 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: true } } }
I20240501 14:04:56.383194 1449 raft_consensus.cc:2932] T 14e0e4808438484688605ff2d6d9258b P eb004c2ac9d244f89b61448584d4115b [term 1 FOLLOWER]: Committing config change with OpId 1.365: config changed from index -1 to 365, NON_VOTER 1be9932ac665476aa1f5ba5b69c73dfa (127.29.124.136) added. New config: { opid_index: 365 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: true } } }
W20240501 14:04:56.384588 1259 consensus_peers.cc:485] T 14e0e4808438484688605ff2d6d9258b P 05f4ef3d126d43e9be32731e708ff76c -> Peer 1be9932ac665476aa1f5ba5b69c73dfa (127.29.124.136:39469): Couldn't send request to peer 1be9932ac665476aa1f5ba5b69c73dfa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 14e0e4808438484688605ff2d6d9258b. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:04:56.387423 1043 catalog_manager.cc:5484] T 14e0e4808438484688605ff2d6d9258b P 05f4ef3d126d43e9be32731e708ff76c reported cstate change: config changed from index -1 to 365, NON_VOTER 1be9932ac665476aa1f5ba5b69c73dfa (127.29.124.136) added. New cstate: current_term: 1 leader_uuid: "05f4ef3d126d43e9be32731e708ff76c" committed_config { opid_index: 365 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:04:56.398797 2883 consensus_queue.cc:237] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { replace: true } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: true } }
I20240501 14:04:56.401314 2621 raft_consensus.cc:1261] T 4da90afe36fd411c8bfd661e4881f86c P 4e6923f9753a4450ac4343c34c427d1e [term 2 FOLLOWER]: Refusing update from remote peer 752bc4569da647679d5df51641961840: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240501 14:04:56.401978 4249 consensus_queue.cc:1035] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240501 14:04:56.404867 4319 raft_consensus.cc:2932] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840 [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 1be9932ac665476aa1f5ba5b69c73dfa (127.29.124.136) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { replace: true } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: true } } }
I20240501 14:04:56.405495 2621 raft_consensus.cc:2932] T 4da90afe36fd411c8bfd661e4881f86c P 4e6923f9753a4450ac4343c34c427d1e [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index -1 to 3, NON_VOTER 1be9932ac665476aa1f5ba5b69c73dfa (127.29.124.136) added. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { replace: true } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: true } } }
W20240501 14:04:56.408500 2819 consensus_peers.cc:485] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840 -> Peer 1be9932ac665476aa1f5ba5b69c73dfa (127.29.124.136:39469): Couldn't send request to peer 1be9932ac665476aa1f5ba5b69c73dfa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4da90afe36fd411c8bfd661e4881f86c. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:04:56.411109 1043 catalog_manager.cc:5484] T 4da90afe36fd411c8bfd661e4881f86c P 4e6923f9753a4450ac4343c34c427d1e reported cstate change: config changed from index -1 to 3, NON_VOTER 1be9932ac665476aa1f5ba5b69c73dfa (127.29.124.136) added. New cstate: current_term: 2 leader_uuid: "752bc4569da647679d5df51641961840" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { replace: true } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: true } } }
I20240501 14:04:56.420910 1323 consensus_queue.cc:237] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 05f4ef3d126d43e9be32731e708ff76c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 380, Committed index: 380, Last appended: 1.380, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 381 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: true } }
I20240501 14:04:56.422904 2883 raft_consensus.cc:1261] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 752bc4569da647679d5df51641961840 [term 1 FOLLOWER]: Refusing update from remote peer 05f4ef3d126d43e9be32731e708ff76c: Log matching property violated. Preceding OpId in replica: term: 1 index: 380. Preceding OpId from leader: term: 1 index: 381. (index mismatch)
W20240501 14:04:56.423048 1259 consensus_peers.cc:485] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 05f4ef3d126d43e9be32731e708ff76c -> Peer 1be9932ac665476aa1f5ba5b69c73dfa (127.29.124.136:39469): Couldn't send request to peer 1be9932ac665476aa1f5ba5b69c73dfa. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3a4ca3f4edcd4bf29d0bab40a8856039. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:04:56.423449 4310 consensus_queue.cc:1035] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 05f4ef3d126d43e9be32731e708ff76c [LEADER]: Connected to new peer: Peer: permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 381, Last known committed idx: 380, Time since last communication: 0.000s
I20240501 14:04:56.425876 3896 raft_consensus.cc:2932] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 05f4ef3d126d43e9be32731e708ff76c [term 1 LEADER]: Committing config change with OpId 1.381: config changed from index 104 to 381, NON_VOTER 1be9932ac665476aa1f5ba5b69c73dfa (127.29.124.136) added. New config: { opid_index: 381 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: true } } }
I20240501 14:04:56.426378 2883 raft_consensus.cc:2932] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 752bc4569da647679d5df51641961840 [term 1 FOLLOWER]: Committing config change with OpId 1.381: config changed from index 104 to 381, NON_VOTER 1be9932ac665476aa1f5ba5b69c73dfa (127.29.124.136) added. New config: { opid_index: 381 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: true } } }
I20240501 14:04:56.430485 1052 catalog_manager.cc:5484] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 05f4ef3d126d43e9be32731e708ff76c reported cstate change: config changed from index 104 to 381, NON_VOTER 1be9932ac665476aa1f5ba5b69c73dfa (127.29.124.136) added. New cstate: current_term: 1 leader_uuid: "05f4ef3d126d43e9be32731e708ff76c" committed_config { opid_index: 381 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:04:56.467039 4201 heartbeater.cc:503] Master 127.29.124.190:46779 was elected leader, sending a full tablet report...
I20240501 14:04:56.790146 4370 ts_tablet_manager.cc:902] T 4da90afe36fd411c8bfd661e4881f86c P 1be9932ac665476aa1f5ba5b69c73dfa: Initiating tablet copy from peer 752bc4569da647679d5df51641961840 (127.29.124.135:42923)
I20240501 14:04:56.791843 4370 tablet_copy_client.cc:308] T 4da90afe36fd411c8bfd661e4881f86c P 1be9932ac665476aa1f5ba5b69c73dfa: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.135:42923
I20240501 14:04:56.797983 2905 tablet_copy_service.cc:140] P 752bc4569da647679d5df51641961840: Received BeginTabletCopySession request for tablet 4da90afe36fd411c8bfd661e4881f86c from peer 1be9932ac665476aa1f5ba5b69c73dfa ({username='slave'} at 127.29.124.136:46991)
I20240501 14:04:56.798178 2905 tablet_copy_service.cc:161] P 752bc4569da647679d5df51641961840: Beginning new tablet copy session on tablet 4da90afe36fd411c8bfd661e4881f86c from peer 1be9932ac665476aa1f5ba5b69c73dfa at {username='slave'} at 127.29.124.136:46991: session id = 1be9932ac665476aa1f5ba5b69c73dfa-4da90afe36fd411c8bfd661e4881f86c
I20240501 14:04:56.801018 2905 tablet_copy_source_session.cc:205] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:04:56.806448 4376 ts_tablet_manager.cc:902] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 1be9932ac665476aa1f5ba5b69c73dfa: Initiating tablet copy from peer 05f4ef3d126d43e9be32731e708ff76c (127.29.124.130:41511)
I20240501 14:04:56.806743 4370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4da90afe36fd411c8bfd661e4881f86c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:56.810168 4376 tablet_copy_client.cc:308] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 1be9932ac665476aa1f5ba5b69c73dfa: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.130:41511
I20240501 14:04:56.816140 1343 tablet_copy_service.cc:140] P 05f4ef3d126d43e9be32731e708ff76c: Received BeginTabletCopySession request for tablet 3a4ca3f4edcd4bf29d0bab40a8856039 from peer 1be9932ac665476aa1f5ba5b69c73dfa ({username='slave'} at 127.29.124.136:45699)
I20240501 14:04:56.816324 1343 tablet_copy_service.cc:161] P 05f4ef3d126d43e9be32731e708ff76c: Beginning new tablet copy session on tablet 3a4ca3f4edcd4bf29d0bab40a8856039 from peer 1be9932ac665476aa1f5ba5b69c73dfa at {username='slave'} at 127.29.124.136:45699: session id = 1be9932ac665476aa1f5ba5b69c73dfa-3a4ca3f4edcd4bf29d0bab40a8856039
I20240501 14:04:56.819097 1343 tablet_copy_source_session.cc:205] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 05f4ef3d126d43e9be32731e708ff76c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:04:56.819592 4370 tablet_copy_client.cc:783] T 4da90afe36fd411c8bfd661e4881f86c P 1be9932ac665476aa1f5ba5b69c73dfa: tablet copy: Starting download of 0 data blocks...
I20240501 14:04:56.819945 4370 tablet_copy_client.cc:647] T 4da90afe36fd411c8bfd661e4881f86c P 1be9932ac665476aa1f5ba5b69c73dfa: tablet copy: Starting download of 1 WAL segments...
I20240501 14:04:56.820664 4376 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a4ca3f4edcd4bf29d0bab40a8856039. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:56.822214 4370 tablet_copy_client.cc:523] T 4da90afe36fd411c8bfd661e4881f86c P 1be9932ac665476aa1f5ba5b69c73dfa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:04:56.826341 4376 tablet_copy_client.cc:783] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 1be9932ac665476aa1f5ba5b69c73dfa: tablet copy: Starting download of 0 data blocks...
I20240501 14:04:56.826342 4370 tablet_bootstrap.cc:492] T 4da90afe36fd411c8bfd661e4881f86c P 1be9932ac665476aa1f5ba5b69c73dfa: Bootstrap starting.
I20240501 14:04:56.826610 4376 tablet_copy_client.cc:647] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 1be9932ac665476aa1f5ba5b69c73dfa: tablet copy: Starting download of 1 WAL segments...
I20240501 14:04:56.829102 4376 tablet_copy_client.cc:523] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 1be9932ac665476aa1f5ba5b69c73dfa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:04:56.832242 4376 tablet_bootstrap.cc:492] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 1be9932ac665476aa1f5ba5b69c73dfa: Bootstrap starting.
I20240501 14:04:56.834600 4370 log.cc:824] T 4da90afe36fd411c8bfd661e4881f86c P 1be9932ac665476aa1f5ba5b69c73dfa: Log is configured to *not* fsync() on all Append() calls
I20240501 14:04:56.838850 4370 tablet_bootstrap.cc:492] T 4da90afe36fd411c8bfd661e4881f86c P 1be9932ac665476aa1f5ba5b69c73dfa: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:04:56.839367 4370 tablet_bootstrap.cc:492] T 4da90afe36fd411c8bfd661e4881f86c P 1be9932ac665476aa1f5ba5b69c73dfa: Bootstrap complete.
I20240501 14:04:56.839830 4370 ts_tablet_manager.cc:1367] T 4da90afe36fd411c8bfd661e4881f86c P 1be9932ac665476aa1f5ba5b69c73dfa: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.006s
I20240501 14:04:56.848793 4370 raft_consensus.cc:348] T 4da90afe36fd411c8bfd661e4881f86c P 1be9932ac665476aa1f5ba5b69c73dfa [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { replace: true } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: true } }
I20240501 14:04:56.849313 4370 raft_consensus.cc:729] T 4da90afe36fd411c8bfd661e4881f86c P 1be9932ac665476aa1f5ba5b69c73dfa [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: 1be9932ac665476aa1f5ba5b69c73dfa, State: Initialized, Role: LEARNER
I20240501 14:04:56.849948 4370 consensus_queue.cc:260] T 4da90afe36fd411c8bfd661e4881f86c P 1be9932ac665476aa1f5ba5b69c73dfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { replace: true } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: true } }
I20240501 14:04:56.851773 4370 ts_tablet_manager.cc:1398] T 4da90afe36fd411c8bfd661e4881f86c P 1be9932ac665476aa1f5ba5b69c73dfa: Time spent starting tablet: real 0.012s user 0.007s sys 0.006s
I20240501 14:04:56.852730 2905 tablet_copy_service.cc:342] P 752bc4569da647679d5df51641961840: Request end of tablet copy session 1be9932ac665476aa1f5ba5b69c73dfa-4da90afe36fd411c8bfd661e4881f86c received from {username='slave'} at 127.29.124.136:46991
I20240501 14:04:56.852893 2905 tablet_copy_service.cc:433] P 752bc4569da647679d5df51641961840: ending tablet copy session 1be9932ac665476aa1f5ba5b69c73dfa-4da90afe36fd411c8bfd661e4881f86c on tablet 4da90afe36fd411c8bfd661e4881f86c with peer 1be9932ac665476aa1f5ba5b69c73dfa
I20240501 14:04:56.888365 4370 ts_tablet_manager.cc:902] T 14e0e4808438484688605ff2d6d9258b P 1be9932ac665476aa1f5ba5b69c73dfa: Initiating tablet copy from peer 05f4ef3d126d43e9be32731e708ff76c (127.29.124.130:41511)
I20240501 14:04:56.889829 4370 tablet_copy_client.cc:308] T 14e0e4808438484688605ff2d6d9258b P 1be9932ac665476aa1f5ba5b69c73dfa: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.130:41511
I20240501 14:04:56.890285 4388 ts_tablet_manager.cc:902] T c586d994d2414073b25a8b3fab6a7046 P 1be9932ac665476aa1f5ba5b69c73dfa: Initiating tablet copy from peer 5d45680f41c84ac39bd97c6f00ac2b06 (127.29.124.129:40925)
I20240501 14:04:56.890640 1343 tablet_copy_service.cc:140] P 05f4ef3d126d43e9be32731e708ff76c: Received BeginTabletCopySession request for tablet 14e0e4808438484688605ff2d6d9258b from peer 1be9932ac665476aa1f5ba5b69c73dfa ({username='slave'} at 127.29.124.136:45699)
I20240501 14:04:56.890799 1343 tablet_copy_service.cc:161] P 05f4ef3d126d43e9be32731e708ff76c: Beginning new tablet copy session on tablet 14e0e4808438484688605ff2d6d9258b from peer 1be9932ac665476aa1f5ba5b69c73dfa at {username='slave'} at 127.29.124.136:45699: session id = 1be9932ac665476aa1f5ba5b69c73dfa-14e0e4808438484688605ff2d6d9258b
I20240501 14:04:56.891892 4388 tablet_copy_client.cc:308] T c586d994d2414073b25a8b3fab6a7046 P 1be9932ac665476aa1f5ba5b69c73dfa: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.129:40925
I20240501 14:04:56.893433 1343 tablet_copy_source_session.cc:205] T 14e0e4808438484688605ff2d6d9258b P 05f4ef3d126d43e9be32731e708ff76c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:04:56.894495 4370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 14e0e4808438484688605ff2d6d9258b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:56.898964 1213 tablet_copy_service.cc:140] P 5d45680f41c84ac39bd97c6f00ac2b06: Received BeginTabletCopySession request for tablet c586d994d2414073b25a8b3fab6a7046 from peer 1be9932ac665476aa1f5ba5b69c73dfa ({username='slave'} at 127.29.124.136:48129)
I20240501 14:04:56.899139 1213 tablet_copy_service.cc:161] P 5d45680f41c84ac39bd97c6f00ac2b06: Beginning new tablet copy session on tablet c586d994d2414073b25a8b3fab6a7046 from peer 1be9932ac665476aa1f5ba5b69c73dfa at {username='slave'} at 127.29.124.136:48129: session id = 1be9932ac665476aa1f5ba5b69c73dfa-c586d994d2414073b25a8b3fab6a7046
I20240501 14:04:56.900015 4370 tablet_copy_client.cc:783] T 14e0e4808438484688605ff2d6d9258b P 1be9932ac665476aa1f5ba5b69c73dfa: tablet copy: Starting download of 0 data blocks...
I20240501 14:04:56.900274 4370 tablet_copy_client.cc:647] T 14e0e4808438484688605ff2d6d9258b P 1be9932ac665476aa1f5ba5b69c73dfa: tablet copy: Starting download of 1 WAL segments...
I20240501 14:04:56.902030 1213 tablet_copy_source_session.cc:205] T c586d994d2414073b25a8b3fab6a7046 P 5d45680f41c84ac39bd97c6f00ac2b06: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:04:56.903426 4388 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c586d994d2414073b25a8b3fab6a7046. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:04:56.904070 4370 tablet_copy_client.cc:523] T 14e0e4808438484688605ff2d6d9258b P 1be9932ac665476aa1f5ba5b69c73dfa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:04:56.908140 4370 tablet_bootstrap.cc:492] T 14e0e4808438484688605ff2d6d9258b P 1be9932ac665476aa1f5ba5b69c73dfa: Bootstrap starting.
I20240501 14:04:56.911157 4388 tablet_copy_client.cc:783] T c586d994d2414073b25a8b3fab6a7046 P 1be9932ac665476aa1f5ba5b69c73dfa: tablet copy: Starting download of 0 data blocks...
I20240501 14:04:56.911446 4388 tablet_copy_client.cc:647] T c586d994d2414073b25a8b3fab6a7046 P 1be9932ac665476aa1f5ba5b69c73dfa: tablet copy: Starting download of 1 WAL segments...
I20240501 14:04:56.914125 4388 tablet_copy_client.cc:523] T c586d994d2414073b25a8b3fab6a7046 P 1be9932ac665476aa1f5ba5b69c73dfa: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:04:56.917534 4388 tablet_bootstrap.cc:492] T c586d994d2414073b25a8b3fab6a7046 P 1be9932ac665476aa1f5ba5b69c73dfa: Bootstrap starting.
I20240501 14:04:57.293994 4156 raft_consensus.cc:1203] T 4da90afe36fd411c8bfd661e4881f86c P 1be9932ac665476aa1f5ba5b69c73dfa [term 2 LEARNER]: Deduplicated request from leader. Original: 2.2->[2.3-2.3] Dedup: 2.3->[]
I20240501 14:04:57.298636 4376 tablet_bootstrap.cc:492] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 1be9932ac665476aa1f5ba5b69c73dfa: Bootstrap replayed 1/1 log segments. Stats: ops{read=381 overwritten=0 applied=381 ignored=0} inserts{seen=1063 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:04:57.299278 4376 tablet_bootstrap.cc:492] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 1be9932ac665476aa1f5ba5b69c73dfa: Bootstrap complete.
I20240501 14:04:57.299669 4376 ts_tablet_manager.cc:1367] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 1be9932ac665476aa1f5ba5b69c73dfa: Time spent bootstrapping tablet: real 0.467s user 0.416s sys 0.028s
I20240501 14:04:57.300851 4376 raft_consensus.cc:348] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 1be9932ac665476aa1f5ba5b69c73dfa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 381 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: true } }
I20240501 14:04:57.301237 4376 raft_consensus.cc:729] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 1be9932ac665476aa1f5ba5b69c73dfa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1be9932ac665476aa1f5ba5b69c73dfa, State: Initialized, Role: LEARNER
I20240501 14:04:57.301568 4376 consensus_queue.cc:260] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 1be9932ac665476aa1f5ba5b69c73dfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 381, Last appended: 1.381, Last appended by leader: 381, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 381 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: true } }
I20240501 14:04:57.302245 4376 ts_tablet_manager.cc:1398] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 1be9932ac665476aa1f5ba5b69c73dfa: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:04:57.303041 1343 tablet_copy_service.cc:342] P 05f4ef3d126d43e9be32731e708ff76c: Request end of tablet copy session 1be9932ac665476aa1f5ba5b69c73dfa-3a4ca3f4edcd4bf29d0bab40a8856039 received from {username='slave'} at 127.29.124.136:45699
I20240501 14:04:57.303195 1343 tablet_copy_service.cc:433] P 05f4ef3d126d43e9be32731e708ff76c: ending tablet copy session 1be9932ac665476aa1f5ba5b69c73dfa-3a4ca3f4edcd4bf29d0bab40a8856039 on tablet 3a4ca3f4edcd4bf29d0bab40a8856039 with peer 1be9932ac665476aa1f5ba5b69c73dfa
I20240501 14:04:57.310837 4156 raft_consensus.cc:1203] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 1be9932ac665476aa1f5ba5b69c73dfa [term 1 LEARNER]: Deduplicated request from leader. Original: 1.380->[1.381-1.381] Dedup: 1.381->[]
I20240501 14:04:57.381728 4388 tablet_bootstrap.cc:492] T c586d994d2414073b25a8b3fab6a7046 P 1be9932ac665476aa1f5ba5b69c73dfa: Bootstrap replayed 1/1 log segments. Stats: ops{read=356 overwritten=0 applied=356 ignored=0} inserts{seen=984 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:04:57.382210 4388 tablet_bootstrap.cc:492] T c586d994d2414073b25a8b3fab6a7046 P 1be9932ac665476aa1f5ba5b69c73dfa: Bootstrap complete.
I20240501 14:04:57.382572 4388 ts_tablet_manager.cc:1367] T c586d994d2414073b25a8b3fab6a7046 P 1be9932ac665476aa1f5ba5b69c73dfa: Time spent bootstrapping tablet: real 0.465s user 0.381s sys 0.037s
I20240501 14:04:57.383399 4388 raft_consensus.cc:348] T c586d994d2414073b25a8b3fab6a7046 P 1be9932ac665476aa1f5ba5b69c73dfa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 356 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: true } }
I20240501 14:04:57.383683 4388 raft_consensus.cc:729] T c586d994d2414073b25a8b3fab6a7046 P 1be9932ac665476aa1f5ba5b69c73dfa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1be9932ac665476aa1f5ba5b69c73dfa, State: Initialized, Role: LEARNER
I20240501 14:04:57.383908 4388 consensus_queue.cc:260] T c586d994d2414073b25a8b3fab6a7046 P 1be9932ac665476aa1f5ba5b69c73dfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 356, Last appended: 1.356, Last appended by leader: 356, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 356 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: true } }
I20240501 14:04:57.384500 4388 ts_tablet_manager.cc:1398] T c586d994d2414073b25a8b3fab6a7046 P 1be9932ac665476aa1f5ba5b69c73dfa: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:04:57.385556 1213 tablet_copy_service.cc:342] P 5d45680f41c84ac39bd97c6f00ac2b06: Request end of tablet copy session 1be9932ac665476aa1f5ba5b69c73dfa-c586d994d2414073b25a8b3fab6a7046 received from {username='slave'} at 127.29.124.136:48129
I20240501 14:04:57.385685 1213 tablet_copy_service.cc:433] P 5d45680f41c84ac39bd97c6f00ac2b06: ending tablet copy session 1be9932ac665476aa1f5ba5b69c73dfa-c586d994d2414073b25a8b3fab6a7046 on tablet c586d994d2414073b25a8b3fab6a7046 with peer 1be9932ac665476aa1f5ba5b69c73dfa
I20240501 14:04:57.425673 4370 tablet_bootstrap.cc:492] T 14e0e4808438484688605ff2d6d9258b P 1be9932ac665476aa1f5ba5b69c73dfa: Bootstrap replayed 1/1 log segments. Stats: ops{read=365 overwritten=0 applied=365 ignored=0} inserts{seen=1037 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:04:57.426059 4370 tablet_bootstrap.cc:492] T 14e0e4808438484688605ff2d6d9258b P 1be9932ac665476aa1f5ba5b69c73dfa: Bootstrap complete.
I20240501 14:04:57.426299 4370 ts_tablet_manager.cc:1367] T 14e0e4808438484688605ff2d6d9258b P 1be9932ac665476aa1f5ba5b69c73dfa: Time spent bootstrapping tablet: real 0.518s user 0.392s sys 0.042s
I20240501 14:04:57.426890 4370 raft_consensus.cc:348] T 14e0e4808438484688605ff2d6d9258b P 1be9932ac665476aa1f5ba5b69c73dfa [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 365 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: true } }
I20240501 14:04:57.427084 4370 raft_consensus.cc:729] T 14e0e4808438484688605ff2d6d9258b P 1be9932ac665476aa1f5ba5b69c73dfa [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1be9932ac665476aa1f5ba5b69c73dfa, State: Initialized, Role: LEARNER
I20240501 14:04:57.427280 4370 consensus_queue.cc:260] T 14e0e4808438484688605ff2d6d9258b P 1be9932ac665476aa1f5ba5b69c73dfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 365, Last appended: 1.365, Last appended by leader: 365, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 365 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: true } }
I20240501 14:04:57.427690 4370 ts_tablet_manager.cc:1398] T 14e0e4808438484688605ff2d6d9258b P 1be9932ac665476aa1f5ba5b69c73dfa: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:04:57.428474 1343 tablet_copy_service.cc:342] P 05f4ef3d126d43e9be32731e708ff76c: Request end of tablet copy session 1be9932ac665476aa1f5ba5b69c73dfa-14e0e4808438484688605ff2d6d9258b received from {username='slave'} at 127.29.124.136:45699
I20240501 14:04:57.428653 1343 tablet_copy_service.cc:433] P 05f4ef3d126d43e9be32731e708ff76c: ending tablet copy session 1be9932ac665476aa1f5ba5b69c73dfa-14e0e4808438484688605ff2d6d9258b on tablet 14e0e4808438484688605ff2d6d9258b with peer 1be9932ac665476aa1f5ba5b69c73dfa
I20240501 14:04:57.474447 4156 raft_consensus.cc:1203] T c586d994d2414073b25a8b3fab6a7046 P 1be9932ac665476aa1f5ba5b69c73dfa [term 1 LEARNER]: Deduplicated request from leader. Original: 1.355->[1.356-1.356] Dedup: 1.356->[]
I20240501 14:04:57.808148 4416 raft_consensus.cc:1050] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 05f4ef3d126d43e9be32731e708ff76c: attempting to promote NON_VOTER 1be9932ac665476aa1f5ba5b69c73dfa to VOTER
I20240501 14:04:57.808784 4416 consensus_queue.cc:237] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 05f4ef3d126d43e9be32731e708ff76c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 381, Committed index: 381, Last appended: 1.381, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 382 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: false } }
I20240501 14:04:57.810920 2883 raft_consensus.cc:1261] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 752bc4569da647679d5df51641961840 [term 1 FOLLOWER]: Refusing update from remote peer 05f4ef3d126d43e9be32731e708ff76c: Log matching property violated. Preceding OpId in replica: term: 1 index: 381. Preceding OpId from leader: term: 1 index: 382. (index mismatch)
I20240501 14:04:57.810945 4156 raft_consensus.cc:1261] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 1be9932ac665476aa1f5ba5b69c73dfa [term 1 LEARNER]: Refusing update from remote peer 05f4ef3d126d43e9be32731e708ff76c: Log matching property violated. Preceding OpId in replica: term: 1 index: 381. Preceding OpId from leader: term: 1 index: 382. (index mismatch)
I20240501 14:04:57.811561 4329 consensus_queue.cc:1035] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 05f4ef3d126d43e9be32731e708ff76c [LEADER]: Connected to new peer: Peer: permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 382, Last known committed idx: 381, Time since last communication: 0.000s
I20240501 14:04:57.811889 4461 consensus_queue.cc:1035] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 05f4ef3d126d43e9be32731e708ff76c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 382, Last known committed idx: 381, Time since last communication: 0.000s
I20240501 14:04:57.814311 4329 raft_consensus.cc:2932] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 05f4ef3d126d43e9be32731e708ff76c [term 1 LEADER]: Committing config change with OpId 1.382: config changed from index 381 to 382, 1be9932ac665476aa1f5ba5b69c73dfa (127.29.124.136) changed from NON_VOTER to VOTER. New config: { opid_index: 382 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: false } } }
I20240501 14:04:57.814895 2883 raft_consensus.cc:2932] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 752bc4569da647679d5df51641961840 [term 1 FOLLOWER]: Committing config change with OpId 1.382: config changed from index 381 to 382, 1be9932ac665476aa1f5ba5b69c73dfa (127.29.124.136) changed from NON_VOTER to VOTER. New config: { opid_index: 382 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: false } } }
I20240501 14:04:57.816257 4156 raft_consensus.cc:2932] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 1be9932ac665476aa1f5ba5b69c73dfa [term 1 FOLLOWER]: Committing config change with OpId 1.382: config changed from index 381 to 382, 1be9932ac665476aa1f5ba5b69c73dfa (127.29.124.136) changed from NON_VOTER to VOTER. New config: { opid_index: 382 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: false } } }
I20240501 14:04:57.819056 1050 catalog_manager.cc:5484] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 05f4ef3d126d43e9be32731e708ff76c reported cstate change: config changed from index 381 to 382, 1be9932ac665476aa1f5ba5b69c73dfa (127.29.124.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "05f4ef3d126d43e9be32731e708ff76c" committed_config { opid_index: 382 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:04:57.829401 4465 raft_consensus.cc:1050] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840: attempting to promote NON_VOTER 1be9932ac665476aa1f5ba5b69c73dfa to VOTER
I20240501 14:04:57.830302 4465 consensus_queue.cc:237] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { replace: true } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: false } }
I20240501 14:04:57.832614 2621 raft_consensus.cc:1261] T 4da90afe36fd411c8bfd661e4881f86c P 4e6923f9753a4450ac4343c34c427d1e [term 2 FOLLOWER]: Refusing update from remote peer 752bc4569da647679d5df51641961840: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:04:57.833127 4464 consensus_queue.cc:1035] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:04:57.834431 4156 raft_consensus.cc:1261] T 4da90afe36fd411c8bfd661e4881f86c P 1be9932ac665476aa1f5ba5b69c73dfa [term 2 LEARNER]: Refusing update from remote peer 752bc4569da647679d5df51641961840: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240501 14:04:57.834914 4319 consensus_queue.cc:1035] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:04:57.837466 4465 raft_consensus.cc:2932] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840 [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 1be9932ac665476aa1f5ba5b69c73dfa (127.29.124.136) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { replace: true } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: false } } }
I20240501 14:04:57.838047 2621 raft_consensus.cc:2932] T 4da90afe36fd411c8bfd661e4881f86c P 4e6923f9753a4450ac4343c34c427d1e [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 1be9932ac665476aa1f5ba5b69c73dfa (127.29.124.136) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { replace: true } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: false } } }
I20240501 14:04:57.842192 1052 catalog_manager.cc:5484] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840 reported cstate change: config changed from index 3 to 4, 1be9932ac665476aa1f5ba5b69c73dfa (127.29.124.136) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "752bc4569da647679d5df51641961840" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:04:57.845237 4156 raft_consensus.cc:2932] T 4da90afe36fd411c8bfd661e4881f86c P 1be9932ac665476aa1f5ba5b69c73dfa [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, 1be9932ac665476aa1f5ba5b69c73dfa (127.29.124.136) changed from NON_VOTER to VOTER. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { replace: true } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: false } } }
I20240501 14:04:57.864564 2883 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4da90afe36fd411c8bfd661e4881f86c"
dest_uuid: "752bc4569da647679d5df51641961840"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47834
I20240501 14:04:57.864797 2883 raft_consensus.cc:595] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840 [term 2 LEADER]: Received request to transfer leadership
I20240501 14:04:57.876459 4156 raft_consensus.cc:1203] T 14e0e4808438484688605ff2d6d9258b P 1be9932ac665476aa1f5ba5b69c73dfa [term 1 LEARNER]: Deduplicated request from leader. Original: 1.364->[1.365-1.365] Dedup: 1.365->[]
I20240501 14:04:57.894600 1323 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3a4ca3f4edcd4bf29d0bab40a8856039"
dest_uuid: "05f4ef3d126d43e9be32731e708ff76c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60924
I20240501 14:04:57.894778 1323 raft_consensus.cc:595] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 05f4ef3d126d43e9be32731e708ff76c [term 1 LEADER]: Received request to transfer leadership
I20240501 14:04:57.998019 4439 raft_consensus.cc:1050] T c586d994d2414073b25a8b3fab6a7046 P 5d45680f41c84ac39bd97c6f00ac2b06: attempting to promote NON_VOTER 1be9932ac665476aa1f5ba5b69c73dfa to VOTER
I20240501 14:04:57.998646 4439 consensus_queue.cc:237] T c586d994d2414073b25a8b3fab6a7046 P 5d45680f41c84ac39bd97c6f00ac2b06 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 356, Committed index: 356, Last appended: 1.356, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 357 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: false } }
I20240501 14:04:58.000536 4156 raft_consensus.cc:1261] T c586d994d2414073b25a8b3fab6a7046 P 1be9932ac665476aa1f5ba5b69c73dfa [term 1 LEARNER]: Refusing update from remote peer 5d45680f41c84ac39bd97c6f00ac2b06: Log matching property violated. Preceding OpId in replica: term: 1 index: 356. Preceding OpId from leader: term: 1 index: 357. (index mismatch)
I20240501 14:04:58.000774 2487 raft_consensus.cc:1261] T c586d994d2414073b25a8b3fab6a7046 P 953c332688d14831b3bc7cfd895e06e9 [term 1 FOLLOWER]: Refusing update from remote peer 5d45680f41c84ac39bd97c6f00ac2b06: Log matching property violated. Preceding OpId in replica: term: 1 index: 356. Preceding OpId from leader: term: 1 index: 357. (index mismatch)
I20240501 14:04:58.001129 4439 consensus_queue.cc:1035] T c586d994d2414073b25a8b3fab6a7046 P 5d45680f41c84ac39bd97c6f00ac2b06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 357, Last known committed idx: 356, Time since last communication: 0.000s
I20240501 14:04:58.001488 4504 consensus_queue.cc:1035] T c586d994d2414073b25a8b3fab6a7046 P 5d45680f41c84ac39bd97c6f00ac2b06 [LEADER]: Connected to new peer: Peer: permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 357, Last known committed idx: 356, Time since last communication: 0.000s
I20240501 14:04:58.004016 4439 raft_consensus.cc:2932] T c586d994d2414073b25a8b3fab6a7046 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 LEADER]: Committing config change with OpId 1.357: config changed from index 356 to 357, 1be9932ac665476aa1f5ba5b69c73dfa (127.29.124.136) changed from NON_VOTER to VOTER. New config: { opid_index: 357 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: false } } }
I20240501 14:04:58.004863 2487 raft_consensus.cc:2932] T c586d994d2414073b25a8b3fab6a7046 P 953c332688d14831b3bc7cfd895e06e9 [term 1 FOLLOWER]: Committing config change with OpId 1.357: config changed from index 356 to 357, 1be9932ac665476aa1f5ba5b69c73dfa (127.29.124.136) changed from NON_VOTER to VOTER. New config: { opid_index: 357 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: false } } }
I20240501 14:04:58.008961 4156 raft_consensus.cc:2932] T c586d994d2414073b25a8b3fab6a7046 P 1be9932ac665476aa1f5ba5b69c73dfa [term 1 FOLLOWER]: Committing config change with OpId 1.357: config changed from index 356 to 357, 1be9932ac665476aa1f5ba5b69c73dfa (127.29.124.136) changed from NON_VOTER to VOTER. New config: { opid_index: 357 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: false } } }
I20240501 14:04:58.009384 1052 catalog_manager.cc:5484] T c586d994d2414073b25a8b3fab6a7046 P 5d45680f41c84ac39bd97c6f00ac2b06 reported cstate change: config changed from index 356 to 357, 1be9932ac665476aa1f5ba5b69c73dfa (127.29.124.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" committed_config { opid_index: 357 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:04:58.121083 2883 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "4da90afe36fd411c8bfd661e4881f86c"
dest_uuid: "752bc4569da647679d5df51641961840"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47844
I20240501 14:04:58.121290 2883 raft_consensus.cc:595] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840 [term 2 LEADER]: Received request to transfer leadership
I20240501 14:04:58.138447 1192 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c586d994d2414073b25a8b3fab6a7046"
dest_uuid: "5d45680f41c84ac39bd97c6f00ac2b06"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35988
I20240501 14:04:58.138656 1192 raft_consensus.cc:595] T c586d994d2414073b25a8b3fab6a7046 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:04:58.155555 1323 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "3a4ca3f4edcd4bf29d0bab40a8856039"
dest_uuid: "05f4ef3d126d43e9be32731e708ff76c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:60942
I20240501 14:04:58.155750 1323 raft_consensus.cc:595] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 05f4ef3d126d43e9be32731e708ff76c [term 1 LEADER]: Received request to transfer leadership
I20240501 14:04:58.237869 4464 raft_consensus.cc:979] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840: : Instructing follower 4e6923f9753a4450ac4343c34c427d1e to start an election
I20240501 14:04:58.238072 4464 raft_consensus.cc:1067] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840 [term 2 LEADER]: Signalling peer 4e6923f9753a4450ac4343c34c427d1e to start an election
I20240501 14:04:58.238794 2621 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "4da90afe36fd411c8bfd661e4881f86c"
dest_uuid: "4e6923f9753a4450ac4343c34c427d1e"
from {username='slave'} at 127.29.124.135:44923
I20240501 14:04:58.239017 2621 raft_consensus.cc:482] T 4da90afe36fd411c8bfd661e4881f86c P 4e6923f9753a4450ac4343c34c427d1e [term 2 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:04:58.239136 2621 raft_consensus.cc:3037] T 4da90afe36fd411c8bfd661e4881f86c P 4e6923f9753a4450ac4343c34c427d1e [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:04:58.241001 2621 raft_consensus.cc:504] T 4da90afe36fd411c8bfd661e4881f86c P 4e6923f9753a4450ac4343c34c427d1e [term 3 FOLLOWER]: Starting forced leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { replace: true } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: false } }
I20240501 14:04:58.241822 2621 leader_election.cc:288] T 4da90afe36fd411c8bfd661e4881f86c P 4e6923f9753a4450ac4343c34c427d1e [CANDIDATE]: Term 3 election: Requested vote from peers 752bc4569da647679d5df51641961840 (127.29.124.135:42923), 1be9932ac665476aa1f5ba5b69c73dfa (127.29.124.136:39469)
I20240501 14:04:58.242372 2883 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4da90afe36fd411c8bfd661e4881f86c" candidate_uuid: "4e6923f9753a4450ac4343c34c427d1e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "752bc4569da647679d5df51641961840"
I20240501 14:04:58.242605 2883 raft_consensus.cc:3032] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840 [term 2 LEADER]: Stepping down as leader of term 2
I20240501 14:04:58.242697 2883 raft_consensus.cc:729] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 752bc4569da647679d5df51641961840, State: Running, Role: LEADER
I20240501 14:04:58.242995 2883 consensus_queue.cc:260] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { replace: true } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: false } }
I20240501 14:04:58.243343 2883 raft_consensus.cc:3037] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:04:58.245160 2883 raft_consensus.cc:2454] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e6923f9753a4450ac4343c34c427d1e in term 3.
I20240501 14:04:58.245630 2555 leader_election.cc:302] T 4da90afe36fd411c8bfd661e4881f86c P 4e6923f9753a4450ac4343c34c427d1e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4e6923f9753a4450ac4343c34c427d1e, 752bc4569da647679d5df51641961840; no voters:
I20240501 14:04:58.245918 4073 raft_consensus.cc:2781] T 4da90afe36fd411c8bfd661e4881f86c P 4e6923f9753a4450ac4343c34c427d1e [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:04:58.246344 4073 raft_consensus.cc:686] T 4da90afe36fd411c8bfd661e4881f86c P 4e6923f9753a4450ac4343c34c427d1e [term 3 LEADER]: Becoming Leader. State: Replica: 4e6923f9753a4450ac4343c34c427d1e, State: Running, Role: LEADER
I20240501 14:04:58.246723 4073 consensus_queue.cc:237] T 4da90afe36fd411c8bfd661e4881f86c P 4e6923f9753a4450ac4343c34c427d1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { replace: true } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: false } }
I20240501 14:04:58.248600 4156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4da90afe36fd411c8bfd661e4881f86c" candidate_uuid: "4e6923f9753a4450ac4343c34c427d1e" candidate_term: 3 candidate_status { last_received { term: 2 index: 4 } } ignore_live_leader: true dest_uuid: "1be9932ac665476aa1f5ba5b69c73dfa"
I20240501 14:04:58.248930 4156 raft_consensus.cc:3037] T 4da90afe36fd411c8bfd661e4881f86c P 1be9932ac665476aa1f5ba5b69c73dfa [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:04:58.249759 1043 catalog_manager.cc:5484] T 4da90afe36fd411c8bfd661e4881f86c P 4e6923f9753a4450ac4343c34c427d1e reported cstate change: term changed from 2 to 3, leader changed from 752bc4569da647679d5df51641961840 (127.29.124.135) to 4e6923f9753a4450ac4343c34c427d1e (127.29.124.133). New cstate: current_term: 3 leader_uuid: "4e6923f9753a4450ac4343c34c427d1e" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:04:58.251828 4156 raft_consensus.cc:2454] T 4da90afe36fd411c8bfd661e4881f86c P 1be9932ac665476aa1f5ba5b69c73dfa [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4e6923f9753a4450ac4343c34c427d1e in term 3.
I20240501 14:04:58.252830 4329 raft_consensus.cc:979] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 05f4ef3d126d43e9be32731e708ff76c: : Instructing follower 752bc4569da647679d5df51641961840 to start an election
I20240501 14:04:58.252981 4329 raft_consensus.cc:1067] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 05f4ef3d126d43e9be32731e708ff76c [term 1 LEADER]: Signalling peer 752bc4569da647679d5df51641961840 to start an election
I20240501 14:04:58.253654 2883 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "3a4ca3f4edcd4bf29d0bab40a8856039"
dest_uuid: "752bc4569da647679d5df51641961840"
from {username='slave'} at 127.29.124.130:44509
I20240501 14:04:58.253823 2883 raft_consensus.cc:482] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 752bc4569da647679d5df51641961840 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:04:58.253918 2883 raft_consensus.cc:3037] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 752bc4569da647679d5df51641961840 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:04:58.255625 2883 raft_consensus.cc:504] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 752bc4569da647679d5df51641961840 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 382 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: false } }
I20240501 14:04:58.256255 2883 leader_election.cc:288] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 752bc4569da647679d5df51641961840 [CANDIDATE]: Term 2 election: Requested vote from peers 05f4ef3d126d43e9be32731e708ff76c (127.29.124.130:41511), 1be9932ac665476aa1f5ba5b69c73dfa (127.29.124.136:39469)
I20240501 14:04:58.256687 4156 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4ca3f4edcd4bf29d0bab40a8856039" candidate_uuid: "752bc4569da647679d5df51641961840" candidate_term: 2 candidate_status { last_received { term: 1 index: 382 } } ignore_live_leader: true dest_uuid: "1be9932ac665476aa1f5ba5b69c73dfa"
I20240501 14:04:58.256685 1323 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a4ca3f4edcd4bf29d0bab40a8856039" candidate_uuid: "752bc4569da647679d5df51641961840" candidate_term: 2 candidate_status { last_received { term: 1 index: 382 } } ignore_live_leader: true dest_uuid: "05f4ef3d126d43e9be32731e708ff76c"
I20240501 14:04:58.256855 1323 raft_consensus.cc:3032] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 05f4ef3d126d43e9be32731e708ff76c [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:04:58.256884 4156 raft_consensus.cc:3037] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 1be9932ac665476aa1f5ba5b69c73dfa [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:04:58.256953 1323 raft_consensus.cc:729] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 05f4ef3d126d43e9be32731e708ff76c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 05f4ef3d126d43e9be32731e708ff76c, State: Running, Role: LEADER
I20240501 14:04:58.257220 1323 consensus_queue.cc:260] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 05f4ef3d126d43e9be32731e708ff76c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 382, Committed index: 382, Last appended: 1.382, Last appended by leader: 382, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 382 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: false } }
I20240501 14:04:58.257534 1323 raft_consensus.cc:3037] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 05f4ef3d126d43e9be32731e708ff76c [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:04:58.258749 4156 raft_consensus.cc:2454] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 1be9932ac665476aa1f5ba5b69c73dfa [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 752bc4569da647679d5df51641961840 in term 2.
I20240501 14:04:58.259099 2819 leader_election.cc:302] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 752bc4569da647679d5df51641961840 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1be9932ac665476aa1f5ba5b69c73dfa, 752bc4569da647679d5df51641961840; no voters:
I20240501 14:04:58.259375 4464 raft_consensus.cc:2781] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 752bc4569da647679d5df51641961840 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:04:58.259477 1323 raft_consensus.cc:2454] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 05f4ef3d126d43e9be32731e708ff76c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 752bc4569da647679d5df51641961840 in term 2.
I20240501 14:04:58.259553 4464 raft_consensus.cc:686] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 752bc4569da647679d5df51641961840 [term 2 LEADER]: Becoming Leader. State: Replica: 752bc4569da647679d5df51641961840, State: Running, Role: LEADER
I20240501 14:04:58.259846 4464 consensus_queue.cc:237] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 752bc4569da647679d5df51641961840 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 382, Committed index: 382, Last appended: 1.382, Last appended by leader: 382, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 382 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: false } }
I20240501 14:04:58.262301 1043 catalog_manager.cc:5484] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 752bc4569da647679d5df51641961840 reported cstate change: term changed from 1 to 2, leader changed from 05f4ef3d126d43e9be32731e708ff76c (127.29.124.130) to 752bc4569da647679d5df51641961840 (127.29.124.135). New cstate: current_term: 2 leader_uuid: "752bc4569da647679d5df51641961840" committed_config { opid_index: 382 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:04:58.389844 1192 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c586d994d2414073b25a8b3fab6a7046"
dest_uuid: "5d45680f41c84ac39bd97c6f00ac2b06"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35998
I20240501 14:04:58.390048 1192 raft_consensus.cc:595] T c586d994d2414073b25a8b3fab6a7046 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:04:58.406972 4571 raft_consensus.cc:1050] T 14e0e4808438484688605ff2d6d9258b P 05f4ef3d126d43e9be32731e708ff76c: attempting to promote NON_VOTER 1be9932ac665476aa1f5ba5b69c73dfa to VOTER
I20240501 14:04:58.407728 4571 consensus_queue.cc:237] T 14e0e4808438484688605ff2d6d9258b P 05f4ef3d126d43e9be32731e708ff76c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 365, Committed index: 365, Last appended: 1.365, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 366 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: false } }
I20240501 14:04:58.409894 4156 raft_consensus.cc:1261] T 14e0e4808438484688605ff2d6d9258b P 1be9932ac665476aa1f5ba5b69c73dfa [term 1 LEARNER]: Refusing update from remote peer 05f4ef3d126d43e9be32731e708ff76c: Log matching property violated. Preceding OpId in replica: term: 1 index: 365. Preceding OpId from leader: term: 1 index: 366. (index mismatch)
I20240501 14:04:58.410141 1449 raft_consensus.cc:1261] T 14e0e4808438484688605ff2d6d9258b P eb004c2ac9d244f89b61448584d4115b [term 1 FOLLOWER]: Refusing update from remote peer 05f4ef3d126d43e9be32731e708ff76c: Log matching property violated. Preceding OpId in replica: term: 1 index: 365. Preceding OpId from leader: term: 1 index: 366. (index mismatch)
I20240501 14:04:58.410524 4571 consensus_queue.cc:1035] T 14e0e4808438484688605ff2d6d9258b P 05f4ef3d126d43e9be32731e708ff76c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 366, Last known committed idx: 365, Time since last communication: 0.000s
I20240501 14:04:58.410810 4416 consensus_queue.cc:1035] T 14e0e4808438484688605ff2d6d9258b P 05f4ef3d126d43e9be32731e708ff76c [LEADER]: Connected to new peer: Peer: permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 366, Last known committed idx: 365, Time since last communication: 0.000s
I20240501 14:04:58.414206 4329 raft_consensus.cc:2932] T 14e0e4808438484688605ff2d6d9258b P 05f4ef3d126d43e9be32731e708ff76c [term 1 LEADER]: Committing config change with OpId 1.366: config changed from index 365 to 366, 1be9932ac665476aa1f5ba5b69c73dfa (127.29.124.136) changed from NON_VOTER to VOTER. New config: { opid_index: 366 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: false } } }
I20240501 14:04:58.414817 1449 raft_consensus.cc:2932] T 14e0e4808438484688605ff2d6d9258b P eb004c2ac9d244f89b61448584d4115b [term 1 FOLLOWER]: Committing config change with OpId 1.366: config changed from index 365 to 366, 1be9932ac665476aa1f5ba5b69c73dfa (127.29.124.136) changed from NON_VOTER to VOTER. New config: { opid_index: 366 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: false } } }
I20240501 14:04:58.417384 4156 raft_consensus.cc:2932] T 14e0e4808438484688605ff2d6d9258b P 1be9932ac665476aa1f5ba5b69c73dfa [term 1 FOLLOWER]: Committing config change with OpId 1.366: config changed from index 365 to 366, 1be9932ac665476aa1f5ba5b69c73dfa (127.29.124.136) changed from NON_VOTER to VOTER. New config: { opid_index: 366 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: false } } }
I20240501 14:04:58.418255 1043 catalog_manager.cc:5484] T 14e0e4808438484688605ff2d6d9258b P 05f4ef3d126d43e9be32731e708ff76c reported cstate change: config changed from index 365 to 366, 1be9932ac665476aa1f5ba5b69c73dfa (127.29.124.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "05f4ef3d126d43e9be32731e708ff76c" committed_config { opid_index: 366 OBSOLETE_local: false peers { permanent_uuid: "eb004c2ac9d244f89b61448584d4115b" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 40225 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:04:58.423528 1323 consensus_queue.cc:237] T 14e0e4808438484688605ff2d6d9258b P 05f4ef3d126d43e9be32731e708ff76c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 366, Committed index: 366, Last appended: 1.366, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 367 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: false } }
I20240501 14:04:58.423699 4504 raft_consensus.cc:979] T c586d994d2414073b25a8b3fab6a7046 P 5d45680f41c84ac39bd97c6f00ac2b06: : Instructing follower 1be9932ac665476aa1f5ba5b69c73dfa to start an election
I20240501 14:04:58.423862 4504 raft_consensus.cc:1067] T c586d994d2414073b25a8b3fab6a7046 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 LEADER]: Signalling peer 1be9932ac665476aa1f5ba5b69c73dfa to start an election
I20240501 14:04:58.424535 4156 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c586d994d2414073b25a8b3fab6a7046"
dest_uuid: "1be9932ac665476aa1f5ba5b69c73dfa"
from {username='slave'} at 127.29.124.129:37439
I20240501 14:04:58.424774 4156 raft_consensus.cc:482] T c586d994d2414073b25a8b3fab6a7046 P 1be9932ac665476aa1f5ba5b69c73dfa [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:04:58.424906 4156 raft_consensus.cc:3037] T c586d994d2414073b25a8b3fab6a7046 P 1be9932ac665476aa1f5ba5b69c73dfa [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:04:58.425061 4155 raft_consensus.cc:1261] T 14e0e4808438484688605ff2d6d9258b P 1be9932ac665476aa1f5ba5b69c73dfa [term 1 FOLLOWER]: Refusing update from remote peer 05f4ef3d126d43e9be32731e708ff76c: Log matching property violated. Preceding OpId in replica: term: 1 index: 366. Preceding OpId from leader: term: 1 index: 367. (index mismatch)
I20240501 14:04:58.425532 4329 consensus_queue.cc:1035] T 14e0e4808438484688605ff2d6d9258b P 05f4ef3d126d43e9be32731e708ff76c [LEADER]: Connected to new peer: Peer: permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 367, Last known committed idx: 366, Time since last communication: 0.000s
I20240501 14:04:58.426940 4156 raft_consensus.cc:504] T c586d994d2414073b25a8b3fab6a7046 P 1be9932ac665476aa1f5ba5b69c73dfa [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 357 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: false } }
I20240501 14:04:58.427373 4571 raft_consensus.cc:2932] T 14e0e4808438484688605ff2d6d9258b P 05f4ef3d126d43e9be32731e708ff76c [term 1 LEADER]: Committing config change with OpId 1.367: config changed from index 366 to 367, VOTER eb004c2ac9d244f89b61448584d4115b (127.29.124.131) evicted. New config: { opid_index: 367 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: false } } }
I20240501 14:04:58.427924 4155 raft_consensus.cc:2932] T 14e0e4808438484688605ff2d6d9258b P 1be9932ac665476aa1f5ba5b69c73dfa [term 1 FOLLOWER]: Committing config change with OpId 1.367: config changed from index 366 to 367, VOTER eb004c2ac9d244f89b61448584d4115b (127.29.124.131) evicted. New config: { opid_index: 367 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: false } } }
I20240501 14:04:58.428881 1192 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c586d994d2414073b25a8b3fab6a7046" candidate_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" candidate_term: 2 candidate_status { last_received { term: 1 index: 357 } } ignore_live_leader: true dest_uuid: "5d45680f41c84ac39bd97c6f00ac2b06"
I20240501 14:04:58.429104 1192 raft_consensus.cc:3032] T c586d994d2414073b25a8b3fab6a7046 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:04:58.429190 1192 raft_consensus.cc:729] T c586d994d2414073b25a8b3fab6a7046 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 5d45680f41c84ac39bd97c6f00ac2b06, State: Running, Role: LEADER
I20240501 14:04:58.429491 1192 consensus_queue.cc:260] T c586d994d2414073b25a8b3fab6a7046 P 5d45680f41c84ac39bd97c6f00ac2b06 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 357, Committed index: 357, Last appended: 1.357, Last appended by leader: 357, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 357 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: false } }
I20240501 14:04:58.429821 1192 raft_consensus.cc:3037] T c586d994d2414073b25a8b3fab6a7046 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:04:58.430831 1038 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 14e0e4808438484688605ff2d6d9258b with cas_config_opid_index 366: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:04:58.431803 1043 catalog_manager.cc:5484] T 14e0e4808438484688605ff2d6d9258b P 1be9932ac665476aa1f5ba5b69c73dfa reported cstate change: config changed from index 366 to 367, VOTER eb004c2ac9d244f89b61448584d4115b (127.29.124.131) evicted. New cstate: current_term: 1 leader_uuid: "05f4ef3d126d43e9be32731e708ff76c" committed_config { opid_index: 367 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: false } } }
I20240501 14:04:58.432247 1192 raft_consensus.cc:2454] T c586d994d2414073b25a8b3fab6a7046 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1be9932ac665476aa1f5ba5b69c73dfa in term 2.
I20240501 14:04:58.432757 4089 leader_election.cc:302] T c586d994d2414073b25a8b3fab6a7046 P 1be9932ac665476aa1f5ba5b69c73dfa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1be9932ac665476aa1f5ba5b69c73dfa, 5d45680f41c84ac39bd97c6f00ac2b06; no voters:
I20240501 14:04:58.433107 4382 raft_consensus.cc:2781] T c586d994d2414073b25a8b3fab6a7046 P 1be9932ac665476aa1f5ba5b69c73dfa [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:04:58.436305 1432 tablet_service.cc:1514] Processing DeleteTablet for tablet 14e0e4808438484688605ff2d6d9258b with delete_type TABLET_DATA_TOMBSTONED (TS eb004c2ac9d244f89b61448584d4115b not found in new config with opid_index 367) from {username='slave'} at 127.0.0.1:38992
I20240501 14:04:58.437870 4579 tablet_replica.cc:330] T 14e0e4808438484688605ff2d6d9258b P eb004c2ac9d244f89b61448584d4115b: stopping tablet replica
I20240501 14:04:58.438140 4579 raft_consensus.cc:2229] T 14e0e4808438484688605ff2d6d9258b P eb004c2ac9d244f89b61448584d4115b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:04:58.438253 4156 leader_election.cc:288] T c586d994d2414073b25a8b3fab6a7046 P 1be9932ac665476aa1f5ba5b69c73dfa [CANDIDATE]: Term 2 election: Requested vote from peers 5d45680f41c84ac39bd97c6f00ac2b06 (127.29.124.129:40925), 953c332688d14831b3bc7cfd895e06e9 (127.29.124.132:34305)
I20240501 14:04:58.438477 4579 raft_consensus.cc:2258] T 14e0e4808438484688605ff2d6d9258b P eb004c2ac9d244f89b61448584d4115b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:04:58.441991 4579 ts_tablet_manager.cc:1875] T 14e0e4808438484688605ff2d6d9258b P eb004c2ac9d244f89b61448584d4115b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:04:58.445853 4382 raft_consensus.cc:686] T c586d994d2414073b25a8b3fab6a7046 P 1be9932ac665476aa1f5ba5b69c73dfa [term 2 LEADER]: Becoming Leader. State: Replica: 1be9932ac665476aa1f5ba5b69c73dfa, State: Running, Role: LEADER
I20240501 14:04:58.446434 4382 consensus_queue.cc:237] T c586d994d2414073b25a8b3fab6a7046 P 1be9932ac665476aa1f5ba5b69c73dfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 357, Committed index: 357, Last appended: 1.357, Last appended by leader: 357, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 357 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: false } }
I20240501 14:04:58.447301 4579 ts_tablet_manager.cc:1888] T 14e0e4808438484688605ff2d6d9258b P eb004c2ac9d244f89b61448584d4115b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.366
I20240501 14:04:58.447451 4579 log.cc:1196] T 14e0e4808438484688605ff2d6d9258b P eb004c2ac9d244f89b61448584d4115b: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/wal/wals/14e0e4808438484688605ff2d6d9258b
I20240501 14:04:58.449568 1039 catalog_manager.cc:4830] TS eb004c2ac9d244f89b61448584d4115b (127.29.124.131:40225): tablet 14e0e4808438484688605ff2d6d9258b (table rebalance_test_table_3 [id=16c6cbe8fef84c88b36c2d1809ebe48b]) successfully deleted
I20240501 14:04:58.449990 1043 catalog_manager.cc:5484] T c586d994d2414073b25a8b3fab6a7046 P 1be9932ac665476aa1f5ba5b69c73dfa reported cstate change: term changed from 1 to 2, leader changed from 5d45680f41c84ac39bd97c6f00ac2b06 (127.29.124.129) to 1be9932ac665476aa1f5ba5b69c73dfa (127.29.124.136). New cstate: current_term: 2 leader_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" committed_config { opid_index: 357 OBSOLETE_local: false peers { permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:04:58.458904 2487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c586d994d2414073b25a8b3fab6a7046" candidate_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" candidate_term: 2 candidate_status { last_received { term: 1 index: 357 } } ignore_live_leader: true dest_uuid: "953c332688d14831b3bc7cfd895e06e9"
I20240501 14:04:58.459105 2487 raft_consensus.cc:3037] T c586d994d2414073b25a8b3fab6a7046 P 953c332688d14831b3bc7cfd895e06e9 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:04:58.460896 2487 raft_consensus.cc:2454] T c586d994d2414073b25a8b3fab6a7046 P 953c332688d14831b3bc7cfd895e06e9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1be9932ac665476aa1f5ba5b69c73dfa in term 2.
I20240501 14:04:58.700909 2883 raft_consensus.cc:1261] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840 [term 3 FOLLOWER]: Refusing update from remote peer 4e6923f9753a4450ac4343c34c427d1e: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240501 14:04:58.701485 4073 consensus_queue.cc:1035] T 4da90afe36fd411c8bfd661e4881f86c P 4e6923f9753a4450ac4343c34c427d1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:04:58.705446 4156 raft_consensus.cc:1261] T 4da90afe36fd411c8bfd661e4881f86c P 1be9932ac665476aa1f5ba5b69c73dfa [term 3 FOLLOWER]: Refusing update from remote peer 4e6923f9753a4450ac4343c34c427d1e: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240501 14:04:58.706020 4073 consensus_queue.cc:1035] T 4da90afe36fd411c8bfd661e4881f86c P 4e6923f9753a4450ac4343c34c427d1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:04:58.706689 1323 raft_consensus.cc:1261] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 05f4ef3d126d43e9be32731e708ff76c [term 2 FOLLOWER]: Refusing update from remote peer 752bc4569da647679d5df51641961840: Log matching property violated. Preceding OpId in replica: term: 1 index: 382. Preceding OpId from leader: term: 2 index: 383. (index mismatch)
I20240501 14:04:58.707873 4464 consensus_queue.cc:1035] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 752bc4569da647679d5df51641961840 [LEADER]: Connected to new peer: Peer: permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 383, Last known committed idx: 382, Time since last communication: 0.000s
I20240501 14:04:58.712292 4156 raft_consensus.cc:1261] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 1be9932ac665476aa1f5ba5b69c73dfa [term 2 FOLLOWER]: Refusing update from remote peer 752bc4569da647679d5df51641961840: Log matching property violated. Preceding OpId in replica: term: 1 index: 382. Preceding OpId from leader: term: 2 index: 383. (index mismatch)
I20240501 14:04:58.713212 4618 consensus_queue.cc:1035] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 752bc4569da647679d5df51641961840 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 383, Last known committed idx: 382, Time since last communication: 0.000s
I20240501 14:04:58.984372 2487 raft_consensus.cc:1261] T c586d994d2414073b25a8b3fab6a7046 P 953c332688d14831b3bc7cfd895e06e9 [term 2 FOLLOWER]: Refusing update from remote peer 1be9932ac665476aa1f5ba5b69c73dfa: Log matching property violated. Preceding OpId in replica: term: 1 index: 357. Preceding OpId from leader: term: 2 index: 358. (index mismatch)
I20240501 14:04:58.985002 4382 consensus_queue.cc:1035] T c586d994d2414073b25a8b3fab6a7046 P 1be9932ac665476aa1f5ba5b69c73dfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 358, Last known committed idx: 357, Time since last communication: 0.000s
I20240501 14:04:58.990638 1192 raft_consensus.cc:1261] T c586d994d2414073b25a8b3fab6a7046 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 2 FOLLOWER]: Refusing update from remote peer 1be9932ac665476aa1f5ba5b69c73dfa: Log matching property violated. Preceding OpId in replica: term: 1 index: 357. Preceding OpId from leader: term: 2 index: 358. (index mismatch)
I20240501 14:04:58.991448 4382 consensus_queue.cc:1035] T c586d994d2414073b25a8b3fab6a7046 P 1be9932ac665476aa1f5ba5b69c73dfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d45680f41c84ac39bd97c6f00ac2b06" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 40925 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 358, Last known committed idx: 357, Time since last communication: 0.000s
I20240501 14:04:59.026010 4156 consensus_queue.cc:237] T c586d994d2414073b25a8b3fab6a7046 P 1be9932ac665476aa1f5ba5b69c73dfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 358, Committed index: 358, Last appended: 2.358, Last appended by leader: 357, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 359 OBSOLETE_local: false peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: false } }
I20240501 14:04:59.027420 2487 raft_consensus.cc:1261] T c586d994d2414073b25a8b3fab6a7046 P 953c332688d14831b3bc7cfd895e06e9 [term 2 FOLLOWER]: Refusing update from remote peer 1be9932ac665476aa1f5ba5b69c73dfa: Log matching property violated. Preceding OpId in replica: term: 2 index: 358. Preceding OpId from leader: term: 2 index: 359. (index mismatch)
I20240501 14:04:59.027858 4382 consensus_queue.cc:1035] T c586d994d2414073b25a8b3fab6a7046 P 1be9932ac665476aa1f5ba5b69c73dfa [LEADER]: Connected to new peer: Peer: permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 359, Last known committed idx: 358, Time since last communication: 0.000s
I20240501 14:04:59.029688 4636 raft_consensus.cc:2932] T c586d994d2414073b25a8b3fab6a7046 P 1be9932ac665476aa1f5ba5b69c73dfa [term 2 LEADER]: Committing config change with OpId 2.359: config changed from index 357 to 359, VOTER 5d45680f41c84ac39bd97c6f00ac2b06 (127.29.124.129) evicted. New config: { opid_index: 359 OBSOLETE_local: false peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: false } } }
I20240501 14:04:59.030225 2487 raft_consensus.cc:2932] T c586d994d2414073b25a8b3fab6a7046 P 953c332688d14831b3bc7cfd895e06e9 [term 2 FOLLOWER]: Committing config change with OpId 2.359: config changed from index 357 to 359, VOTER 5d45680f41c84ac39bd97c6f00ac2b06 (127.29.124.129) evicted. New config: { opid_index: 359 OBSOLETE_local: false peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: false } } }
I20240501 14:04:59.032382 1039 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c586d994d2414073b25a8b3fab6a7046 with cas_config_opid_index 357: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:04:59.033808 1043 catalog_manager.cc:5484] T c586d994d2414073b25a8b3fab6a7046 P 1be9932ac665476aa1f5ba5b69c73dfa reported cstate change: config changed from index 357 to 359, VOTER 5d45680f41c84ac39bd97c6f00ac2b06 (127.29.124.129) evicted. New cstate: current_term: 2 leader_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" committed_config { opid_index: 359 OBSOLETE_local: false peers { permanent_uuid: "953c332688d14831b3bc7cfd895e06e9" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 34305 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:04:59.037750 1172 tablet_service.cc:1514] Processing DeleteTablet for tablet c586d994d2414073b25a8b3fab6a7046 with delete_type TABLET_DATA_TOMBSTONED (TS 5d45680f41c84ac39bd97c6f00ac2b06 not found in new config with opid_index 359) from {username='slave'} at 127.0.0.1:51144
I20240501 14:04:59.038360 4644 tablet_replica.cc:330] T c586d994d2414073b25a8b3fab6a7046 P 5d45680f41c84ac39bd97c6f00ac2b06: stopping tablet replica
I20240501 14:04:59.038569 4644 raft_consensus.cc:2229] T c586d994d2414073b25a8b3fab6a7046 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:04:59.038794 4644 raft_consensus.cc:2258] T c586d994d2414073b25a8b3fab6a7046 P 5d45680f41c84ac39bd97c6f00ac2b06 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:04:59.040105 4644 ts_tablet_manager.cc:1875] T c586d994d2414073b25a8b3fab6a7046 P 5d45680f41c84ac39bd97c6f00ac2b06: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:04:59.044391 4644 ts_tablet_manager.cc:1888] T c586d994d2414073b25a8b3fab6a7046 P 5d45680f41c84ac39bd97c6f00ac2b06: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.358
I20240501 14:04:59.044517 4644 log.cc:1196] T c586d994d2414073b25a8b3fab6a7046 P 5d45680f41c84ac39bd97c6f00ac2b06: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/wal/wals/c586d994d2414073b25a8b3fab6a7046
I20240501 14:04:59.045235 1036 catalog_manager.cc:4830] TS 5d45680f41c84ac39bd97c6f00ac2b06 (127.29.124.129:40925): tablet c586d994d2414073b25a8b3fab6a7046 (table rebalance_test_table_4 [id=6fd9bb23d84f4078a694c2c7a0cb93b2]) successfully deleted
I20240501 14:04:59.203544 2621 consensus_queue.cc:237] T 4da90afe36fd411c8bfd661e4881f86c P 4e6923f9753a4450ac4343c34c427d1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 3.5, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: false } }
I20240501 14:04:59.205179 4156 raft_consensus.cc:1261] T 4da90afe36fd411c8bfd661e4881f86c P 1be9932ac665476aa1f5ba5b69c73dfa [term 3 FOLLOWER]: Refusing update from remote peer 4e6923f9753a4450ac4343c34c427d1e: Log matching property violated. Preceding OpId in replica: term: 3 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20240501 14:04:59.205762 4545 consensus_queue.cc:1035] T 4da90afe36fd411c8bfd661e4881f86c P 4e6923f9753a4450ac4343c34c427d1e [LEADER]: Connected to new peer: Peer: permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:04:59.207796 4670 raft_consensus.cc:2932] T 4da90afe36fd411c8bfd661e4881f86c P 4e6923f9753a4450ac4343c34c427d1e [term 3 LEADER]: Committing config change with OpId 3.6: config changed from index 4 to 6, VOTER 752bc4569da647679d5df51641961840 (127.29.124.135) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: false } } }
I20240501 14:04:59.208374 4156 raft_consensus.cc:2932] T 4da90afe36fd411c8bfd661e4881f86c P 1be9932ac665476aa1f5ba5b69c73dfa [term 3 FOLLOWER]: Committing config change with OpId 3.6: config changed from index 4 to 6, VOTER 752bc4569da647679d5df51641961840 (127.29.124.135) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: false } } }
I20240501 14:04:59.211889 1036 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4da90afe36fd411c8bfd661e4881f86c with cas_config_opid_index 4: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:04:59.213317 1050 catalog_manager.cc:5484] T 4da90afe36fd411c8bfd661e4881f86c P 4e6923f9753a4450ac4343c34c427d1e reported cstate change: config changed from index 4 to 6, VOTER 752bc4569da647679d5df51641961840 (127.29.124.135) evicted. New cstate: current_term: 3 leader_uuid: "4e6923f9753a4450ac4343c34c427d1e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "4e6923f9753a4450ac4343c34c427d1e" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 34675 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:04:59.216619 2864 tablet_service.cc:1514] Processing DeleteTablet for tablet 4da90afe36fd411c8bfd661e4881f86c with delete_type TABLET_DATA_TOMBSTONED (TS 752bc4569da647679d5df51641961840 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39966
I20240501 14:04:59.217336 4671 tablet_replica.cc:330] stopping tablet replica
I20240501 14:04:59.217692 4671 raft_consensus.cc:2229] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840 [term 3 FOLLOWER]: Raft consensus shutting down.
I20240501 14:04:59.217909 4671 raft_consensus.cc:2258] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840 [term 3 FOLLOWER]: Raft consensus is shut down!
I20240501 14:04:59.219478 4671 ts_tablet_manager.cc:1875] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:04:59.223765 4671 ts_tablet_manager.cc:1888] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 3.5
I20240501 14:04:59.223898 4671 log.cc:1196] T 4da90afe36fd411c8bfd661e4881f86c P 752bc4569da647679d5df51641961840: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/wal/wals/4da90afe36fd411c8bfd661e4881f86c
I20240501 14:04:59.224592 1038 catalog_manager.cc:4830] TS 752bc4569da647679d5df51641961840 (127.29.124.135:42923): tablet 4da90afe36fd411c8bfd661e4881f86c (table TestTable [id=e0d89fa711ce48dd87e813f93cca2bd4]) successfully deleted
I20240501 14:04:59.327248 2883 consensus_queue.cc:237] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 752bc4569da647679d5df51641961840 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 383, Committed index: 383, Last appended: 2.383, Last appended by leader: 382, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 384 OBSOLETE_local: false peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: false } }
I20240501 14:04:59.328640 4156 raft_consensus.cc:1261] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 1be9932ac665476aa1f5ba5b69c73dfa [term 2 FOLLOWER]: Refusing update from remote peer 752bc4569da647679d5df51641961840: Log matching property violated. Preceding OpId in replica: term: 2 index: 383. Preceding OpId from leader: term: 2 index: 384. (index mismatch)
I20240501 14:04:59.329109 4673 consensus_queue.cc:1035] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 752bc4569da647679d5df51641961840 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 384, Last known committed idx: 383, Time since last communication: 0.000s
I20240501 14:04:59.331007 4672 raft_consensus.cc:2932] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 752bc4569da647679d5df51641961840 [term 2 LEADER]: Committing config change with OpId 2.384: config changed from index 382 to 384, VOTER 05f4ef3d126d43e9be32731e708ff76c (127.29.124.130) evicted. New config: { opid_index: 384 OBSOLETE_local: false peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: false } } }
I20240501 14:04:59.331521 4156 raft_consensus.cc:2932] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 1be9932ac665476aa1f5ba5b69c73dfa [term 2 FOLLOWER]: Committing config change with OpId 2.384: config changed from index 382 to 384, VOTER 05f4ef3d126d43e9be32731e708ff76c (127.29.124.130) evicted. New config: { opid_index: 384 OBSOLETE_local: false peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: false } } }
I20240501 14:04:59.333673 1038 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3a4ca3f4edcd4bf29d0bab40a8856039 with cas_config_opid_index 382: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:04:59.335196 1043 catalog_manager.cc:5484] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 752bc4569da647679d5df51641961840 reported cstate change: config changed from index 382 to 384, VOTER 05f4ef3d126d43e9be32731e708ff76c (127.29.124.130) evicted. New cstate: current_term: 2 leader_uuid: "752bc4569da647679d5df51641961840" committed_config { opid_index: 384 OBSOLETE_local: false peers { permanent_uuid: "752bc4569da647679d5df51641961840" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 42923 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:04:59.338393 1302 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a4ca3f4edcd4bf29d0bab40a8856039 with delete_type TABLET_DATA_TOMBSTONED (TS 05f4ef3d126d43e9be32731e708ff76c not found in new config with opid_index 384) from {username='slave'} at 127.0.0.1:35916
I20240501 14:04:59.338935 4674 tablet_replica.cc:330] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 05f4ef3d126d43e9be32731e708ff76c: stopping tablet replica
I20240501 14:04:59.339118 4674 raft_consensus.cc:2229] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 05f4ef3d126d43e9be32731e708ff76c [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:04:59.339334 4674 raft_consensus.cc:2258] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 05f4ef3d126d43e9be32731e708ff76c [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:04:59.340680 4674 ts_tablet_manager.cc:1875] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 05f4ef3d126d43e9be32731e708ff76c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:04:59.345125 4674 ts_tablet_manager.cc:1888] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 05f4ef3d126d43e9be32731e708ff76c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.383
I20240501 14:04:59.345216 4674 log.cc:1196] T 3a4ca3f4edcd4bf29d0bab40a8856039 P 05f4ef3d126d43e9be32731e708ff76c: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/wal/wals/3a4ca3f4edcd4bf29d0bab40a8856039
I20240501 14:04:59.345883 1038 catalog_manager.cc:4830] TS 05f4ef3d126d43e9be32731e708ff76c (127.29.124.130:41511): tablet 3a4ca3f4edcd4bf29d0bab40a8856039 (table rebalance_test_table_2 [id=9c1d190b2b264f27af23180ab8f30d42]) successfully deleted
I20240501 14:04:59.462309 4136 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:59.466419 1302 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:59.481066 2602 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:04:59.496029 2733 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:04:59.516278 2864 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:04:59.539753 1432 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:04:59.554340 2469 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:04:59.564272 1172 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:04:59.719019 1302 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:04:59.721450 4136 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:04:59.738700 2733 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:04:59.740978 2864 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:04:59.743589 2469 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:04:59.752049 1432 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:04:59.753545 1172 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:04:59.755765 2602 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:05:00.093995 1302 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:00.102144 4136 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:00.111934 2733 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:00.119720 1172 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:05:00.143345 2602 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240501 14:05:00.144454 2864 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240501 14:05:00.184643 2469 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:00.191902 1432 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+----------------------+---------
3e721134e18641eca52cbebd71ec0520 | 127.29.124.190:46779 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.29.124.148:36129 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+----------------------+---------+----------+----------------+-----------------
05f4ef3d126d43e9be32731e708ff76c | 127.29.124.130:41511 | HEALTHY | <none> | 4 | 0
1be9932ac665476aa1f5ba5b69c73dfa | 127.29.124.136:39469 | HEALTHY | <none> | 1 | 0
32064197afde4b79bf388b080c365d88 | 127.29.124.134:37211 | HEALTHY | <none> | 0 | 0
4e6923f9753a4450ac4343c34c427d1e | 127.29.124.133:34675 | HEALTHY | <none> | 3 | 0
5d45680f41c84ac39bd97c6f00ac2b06 | 127.29.124.129:40925 | HEALTHY | <none> | 3 | 0
752bc4569da647679d5df51641961840 | 127.29.124.135:42923 | HEALTHY | <none> | 2 | 0
953c332688d14831b3bc7cfd895e06e9 | 127.29.124.132:34305 | HEALTHY | <none> | 0 | 0
eb004c2ac9d244f89b61448584d4115b | 127.29.124.131:40225 | HEALTHY | <none> | 3 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.29.124.129 | experimental | 127.29.124.129:40925
local_ip_for_outbound_sockets | 127.29.124.130 | experimental | 127.29.124.130:41511
local_ip_for_outbound_sockets | 127.29.124.131 | experimental | 127.29.124.131:40225
local_ip_for_outbound_sockets | 127.29.124.132 | experimental | 127.29.124.132:34305
local_ip_for_outbound_sockets | 127.29.124.133 | experimental | 127.29.124.133:34675
local_ip_for_outbound_sockets | 127.29.124.134 | experimental | 127.29.124.134:37211
local_ip_for_outbound_sockets | 127.29.124.135 | experimental | 127.29.124.135:42923
local_ip_for_outbound_sockets | 127.29.124.136 | experimental | 127.29.124.136:39469
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.29.124.129:40925
server_dump_info_path | /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.29.124.130:41511
server_dump_info_path | /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.29.124.131:40225
server_dump_info_path | /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.29.124.132:34305
server_dump_info_path | /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.29.124.133:34675
server_dump_info_path | /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.29.124.134:37211
server_dump_info_path | /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.29.124.135:42923
server_dump_info_path | /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.RebalanceParamTest.Rebalance_3.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.29.124.136:39469
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+----------------------+-------------------------
builtin_ntp_servers | 127.29.124.148:36129 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 2 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 2 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 4
First Quartile | 4
Median | 4
Third Quartile | 4
Maximum | 4
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 32
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240501 14:05:00.245220 30194 log_verifier.cc:126] Checking tablet 0f11914838ca4139a1c1813fc5334782
I20240501 14:05:00.379590 30194 log_verifier.cc:177] Verified matching terms for 366 ops in tablet 0f11914838ca4139a1c1813fc5334782
I20240501 14:05:00.380057 30194 log_verifier.cc:126] Checking tablet 0f837cac32ad4aaa970124639138c2db
I20240501 14:05:00.508129 30194 log_verifier.cc:177] Verified matching terms for 383 ops in tablet 0f837cac32ad4aaa970124639138c2db
I20240501 14:05:00.508611 30194 log_verifier.cc:126] Checking tablet 1074c96bc2564edca376f95bd195a62c
I20240501 14:05:00.642043 30194 log_verifier.cc:177] Verified matching terms for 404 ops in tablet 1074c96bc2564edca376f95bd195a62c
I20240501 14:05:00.642518 30194 log_verifier.cc:126] Checking tablet 14e026c65736401fb3aec7e6462ae001
I20240501 14:05:00.765746 30194 log_verifier.cc:177] Verified matching terms for 359 ops in tablet 14e026c65736401fb3aec7e6462ae001
I20240501 14:05:00.766487 30194 log_verifier.cc:126] Checking tablet 14e0e4808438484688605ff2d6d9258b
I20240501 14:05:00.898100 30194 log_verifier.cc:177] Verified matching terms for 367 ops in tablet 14e0e4808438484688605ff2d6d9258b
I20240501 14:05:00.898559 30194 log_verifier.cc:126] Checking tablet 3a4ca3f4edcd4bf29d0bab40a8856039
I20240501 14:05:01.060633 30194 log_verifier.cc:177] Verified matching terms for 384 ops in tablet 3a4ca3f4edcd4bf29d0bab40a8856039
I20240501 14:05:01.061111 30194 log_verifier.cc:126] Checking tablet 45d428dc499a4270b56a579f5b3cad83
I20240501 14:05:01.196920 30194 log_verifier.cc:177] Verified matching terms for 376 ops in tablet 45d428dc499a4270b56a579f5b3cad83
I20240501 14:05:01.197424 30194 log_verifier.cc:126] Checking tablet 4c22d4541da544f5b812c6e5e8c2ea40
I20240501 14:05:01.318403 30194 log_verifier.cc:177] Verified matching terms for 357 ops in tablet 4c22d4541da544f5b812c6e5e8c2ea40
I20240501 14:05:01.318843 30194 log_verifier.cc:126] Checking tablet 4da90afe36fd411c8bfd661e4881f86c
I20240501 14:05:01.331676 30194 log_verifier.cc:177] Verified matching terms for 6 ops in tablet 4da90afe36fd411c8bfd661e4881f86c
I20240501 14:05:01.331770 30194 log_verifier.cc:126] Checking tablet 53249c71dab94ff4b179309088f4d1cd
I20240501 14:05:01.457787 30194 log_verifier.cc:177] Verified matching terms for 362 ops in tablet 53249c71dab94ff4b179309088f4d1cd
I20240501 14:05:01.468873 30194 log_verifier.cc:126] Checking tablet 6a5325b7aa824831a5fa056fb4439110
I20240501 14:05:01.583161 30194 log_verifier.cc:177] Verified matching terms for 346 ops in tablet 6a5325b7aa824831a5fa056fb4439110
I20240501 14:05:01.583559 30194 log_verifier.cc:126] Checking tablet 70afcb72943b49a7bea962791b38a91d
I20240501 14:05:01.701597 30194 log_verifier.cc:177] Verified matching terms for 345 ops in tablet 70afcb72943b49a7bea962791b38a91d
I20240501 14:05:01.701992 30194 log_verifier.cc:126] Checking tablet 8979fd4d21814622ba687edb88fa7851
I20240501 14:05:01.843956 30194 log_verifier.cc:177] Verified matching terms for 406 ops in tablet 8979fd4d21814622ba687edb88fa7851
I20240501 14:05:01.844429 30194 log_verifier.cc:126] Checking tablet c586d994d2414073b25a8b3fab6a7046
I20240501 14:05:01.970715 30194 log_verifier.cc:177] Verified matching terms for 359 ops in tablet c586d994d2414073b25a8b3fab6a7046
I20240501 14:05:01.971146 30194 log_verifier.cc:126] Checking tablet db2b3f73d20c4790808bbe0f1d5e489c
I20240501 14:05:02.107647 30194 log_verifier.cc:177] Verified matching terms for 361 ops in tablet db2b3f73d20c4790808bbe0f1d5e489c
I20240501 14:05:02.108052 30194 log_verifier.cc:126] Checking tablet ef358ca6b48646638ec351cb06cf704d
I20240501 14:05:02.223965 30194 log_verifier.cc:177] Verified matching terms for 306 ops in tablet ef358ca6b48646638ec351cb06cf704d
I20240501 14:05:02.231600 30194 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUv7aek/build/asan/bin/kudu pid 1111
I20240501 14:05:02.232797 1233 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:05:02.458796 30194 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUv7aek/build/asan/bin/kudu with pid 1111
I20240501 14:05:02.493456 30194 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUv7aek/build/asan/bin/kudu pid 1241
I20240501 14:05:02.494524 1362 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:05:02.720597 30194 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUv7aek/build/asan/bin/kudu with pid 1241
I20240501 14:05:02.755908 30194 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUv7aek/build/asan/bin/kudu pid 1371
I20240501 14:05:02.756979 1492 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:05:02.975487 30194 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUv7aek/build/asan/bin/kudu with pid 1371
W20240501 14:05:03.014494 2688 connection.cc:535] server connection from 127.29.124.131:49057 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:05:03.014652 2554 connection.cc:535] server connection from 127.29.124.131:51837 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:05:03.015269 30194 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUv7aek/build/asan/bin/kudu pid 2407
I20240501 14:05:03.025187 2529 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:05:03.182129 30194 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUv7aek/build/asan/bin/kudu with pid 2407
W20240501 14:05:03.212730 1037 connection.cc:535] server connection from 127.29.124.132:45253 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:05:03.212695 2555 connection.cc:535] client connection to 127.29.124.132:34305 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:05:03.212996 2555 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:05:03.213299 30194 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUv7aek/build/asan/bin/kudu pid 2538
W20240501 14:05:03.215494 2555 consensus_peers.cc:485] T 6a5325b7aa824831a5fa056fb4439110 P 4e6923f9753a4450ac4343c34c427d1e -> Peer 953c332688d14831b3bc7cfd895e06e9 (127.29.124.132:34305): Couldn't send request to peer 953c332688d14831b3bc7cfd895e06e9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.132:34305: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:03.219513 2662 generic_service.cc:196] Checking for leaks (request via RPC)
W20240501 14:05:03.225416 4091 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.132:34305: connect: Connection refused (error 111)
W20240501 14:05:03.226770 4091 consensus_peers.cc:485] T c586d994d2414073b25a8b3fab6a7046 P 1be9932ac665476aa1f5ba5b69c73dfa -> Peer 953c332688d14831b3bc7cfd895e06e9 (127.29.124.132:34305): Couldn't send request to peer 953c332688d14831b3bc7cfd895e06e9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.132:34305: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:05:03.300523 2819 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.131:40225: connect: Connection refused (error 111)
W20240501 14:05:03.302140 2819 consensus_peers.cc:485] T 0f837cac32ad4aaa970124639138c2db P 752bc4569da647679d5df51641961840 -> Peer eb004c2ac9d244f89b61448584d4115b (127.29.124.131:40225): Couldn't send request to peer eb004c2ac9d244f89b61448584d4115b. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.131:40225: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:03.371549 30194 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUv7aek/build/asan/bin/kudu with pid 2538
W20240501 14:05:03.394409 1037 connection.cc:535] server connection from 127.29.124.133:38581 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:05:03.394551 30194 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUv7aek/build/asan/bin/kudu pid 2671
I20240501 14:05:03.400290 2792 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:05:03.554665 30194 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUv7aek/build/asan/bin/kudu with pid 2671
I20240501 14:05:03.578696 30194 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUv7aek/build/asan/bin/kudu pid 2801
I20240501 14:05:03.584497 2926 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:05:03.748603 30194 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUv7aek/build/asan/bin/kudu with pid 2801
I20240501 14:05:03.773375 30194 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUv7aek/build/asan/bin/kudu pid 4074
I20240501 14:05:03.779034 4196 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:05:03.896598 4812 raft_consensus.cc:482] T 14e0e4808438484688605ff2d6d9258b P 1be9932ac665476aa1f5ba5b69c73dfa [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 05f4ef3d126d43e9be32731e708ff76c)
I20240501 14:05:03.896845 4812 raft_consensus.cc:504] T 14e0e4808438484688605ff2d6d9258b P 1be9932ac665476aa1f5ba5b69c73dfa [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 367 OBSOLETE_local: false peers { permanent_uuid: "05f4ef3d126d43e9be32731e708ff76c" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 41511 } } peers { permanent_uuid: "1be9932ac665476aa1f5ba5b69c73dfa" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 39469 } attrs { promote: false } }
I20240501 14:05:03.897238 30194 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUv7aek/build/asan/bin/kudu with pid 4074
I20240501 14:05:03.914165 30194 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUv7aek/build/asan/bin/kudu pid 1021
I20240501 14:05:03.915203 1082 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:05:04.073103 30194 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUv7aek/build/asan/bin/kudu with pid 1021
2024-05-01T14:05:04Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/3 (46599 ms)
[----------] 1 test from RebalanceParamTest (46599 ms total)
[----------] 1 test from ConcurrentRebalancersTest
[ RUN ] ConcurrentRebalancersTest.TwoConcurrentRebalancers/1
I20240501 14:05:04.147365 30194 test_util.cc:268] Using random seed: -115252330
I20240501 14:05:04.149049 30194 ts_itest-base.cc:112] Starting cluster with:
I20240501 14:05:04.149101 30194 ts_itest-base.cc:113] --------------
I20240501 14:05:04.149121 30194 ts_itest-base.cc:114] 8 tablet servers
I20240501 14:05:04.149147 30194 ts_itest-base.cc:115] 3 replicas per TS
I20240501 14:05:04.149173 30194 ts_itest-base.cc:116] --------------
2024-05-01T14:05:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-01T14:05:04Z Disabled control of system clock
I20240501 14:05:04.195729 30194 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUv7aek/build/asan/bin/kudu
/tmp/dist-test-taskUv7aek/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.124.190:35761
--webserver_interface=127.29.124.190
--webserver_port=0
--builtin_ntp_servers=127.29.124.148:46431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.29.124.190:35761
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240501 14:05:04.517973 4823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:05:04.518285 4823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:05:04.518405 4823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:05:04.525946 4823 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:05:04.526086 4823 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240501 14:05:04.526165 4823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:05:04.526235 4823 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240501 14:05:04.526296 4823 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240501 14:05:04.535729 4823 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.124.148:46431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.29.124.190:35761
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.29.124.190:35761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.29.124.190
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:05:04.537027 4823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:05:04.538753 4823 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:05:04.549005 4829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:05:04.549408 4828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:05:04.549791 4831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:05:04.550360 4823 server_base.cc:1034] running on GCE node
I20240501 14:05:04.551142 4823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:05:04.552537 4823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:05:04.553731 4823 hybrid_clock.cc:648] HybridClock initialized: now 1714572304553676 us; error 47 us; skew 500 ppm
I20240501 14:05:04.554076 4823 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:05:04.556303 4823 webserver.cc:460] Webserver started at http://127.29.124.190:33927/ using document root <none> and password file <none>
I20240501 14:05:04.557022 4823 fs_manager.cc:359] Metadata directory not provided
I20240501 14:05:04.557152 4823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:05:04.557500 4823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:05:04.560063 4823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "f10c6a9c4bad46deb63190d64169ec11"
format_stamp: "Formatted at 2024-05-01 14:05:04 on dist-test-slave-p83k"
I20240501 14:05:04.560812 4823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "f10c6a9c4bad46deb63190d64169ec11"
format_stamp: "Formatted at 2024-05-01 14:05:04 on dist-test-slave-p83k"
I20240501 14:05:04.565905 4823 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240501 14:05:04.569193 4837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:05:04.570648 4823 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 14:05:04.570802 4823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/wal
uuid: "f10c6a9c4bad46deb63190d64169ec11"
format_stamp: "Formatted at 2024-05-01 14:05:04 on dist-test-slave-p83k"
I20240501 14:05:04.570982 4823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:05:04.584560 4823 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:05:04.585394 4823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:05:04.585639 4823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:05:04.601903 4823 rpc_server.cc:307] RPC server started. Bound to: 127.29.124.190:35761
I20240501 14:05:04.601928 4888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.124.190:35761 every 8 connection(s)
I20240501 14:05:04.603260 4823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240501 14:05:04.606998 4889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:04.610564 30194 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUv7aek/build/asan/bin/kudu as pid 4823
I20240501 14:05:04.610721 30194 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240501 14:05:04.618266 4889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f10c6a9c4bad46deb63190d64169ec11: Bootstrap starting.
I20240501 14:05:04.622116 4889 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f10c6a9c4bad46deb63190d64169ec11: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:04.623502 4889 log.cc:824] T 00000000000000000000000000000000 P f10c6a9c4bad46deb63190d64169ec11: Log is configured to *not* fsync() on all Append() calls
I20240501 14:05:04.626519 4889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f10c6a9c4bad46deb63190d64169ec11: No bootstrap required, opened a new log
I20240501 14:05:04.633304 4889 raft_consensus.cc:348] T 00000000000000000000000000000000 P f10c6a9c4bad46deb63190d64169ec11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f10c6a9c4bad46deb63190d64169ec11" member_type: VOTER last_known_addr { host: "127.29.124.190" port: 35761 } }
I20240501 14:05:04.633659 4889 raft_consensus.cc:374] T 00000000000000000000000000000000 P f10c6a9c4bad46deb63190d64169ec11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:04.633741 4889 raft_consensus.cc:729] T 00000000000000000000000000000000 P f10c6a9c4bad46deb63190d64169ec11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f10c6a9c4bad46deb63190d64169ec11, State: Initialized, Role: FOLLOWER
I20240501 14:05:04.634301 4889 consensus_queue.cc:260] T 00000000000000000000000000000000 P f10c6a9c4bad46deb63190d64169ec11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f10c6a9c4bad46deb63190d64169ec11" member_type: VOTER last_known_addr { host: "127.29.124.190" port: 35761 } }
I20240501 14:05:04.634483 4889 raft_consensus.cc:388] T 00000000000000000000000000000000 P f10c6a9c4bad46deb63190d64169ec11 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240501 14:05:04.634573 4889 raft_consensus.cc:482] T 00000000000000000000000000000000 P f10c6a9c4bad46deb63190d64169ec11 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240501 14:05:04.634694 4889 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f10c6a9c4bad46deb63190d64169ec11 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:04.636947 4889 raft_consensus.cc:504] T 00000000000000000000000000000000 P f10c6a9c4bad46deb63190d64169ec11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f10c6a9c4bad46deb63190d64169ec11" member_type: VOTER last_known_addr { host: "127.29.124.190" port: 35761 } }
I20240501 14:05:04.637492 4889 leader_election.cc:302] T 00000000000000000000000000000000 P f10c6a9c4bad46deb63190d64169ec11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f10c6a9c4bad46deb63190d64169ec11; no voters:
I20240501 14:05:04.638037 4889 leader_election.cc:288] T 00000000000000000000000000000000 P f10c6a9c4bad46deb63190d64169ec11 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240501 14:05:04.638178 4894 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f10c6a9c4bad46deb63190d64169ec11 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:05:04.638731 4894 raft_consensus.cc:686] T 00000000000000000000000000000000 P f10c6a9c4bad46deb63190d64169ec11 [term 1 LEADER]: Becoming Leader. State: Replica: f10c6a9c4bad46deb63190d64169ec11, State: Running, Role: LEADER
I20240501 14:05:04.639279 4894 consensus_queue.cc:237] T 00000000000000000000000000000000 P f10c6a9c4bad46deb63190d64169ec11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f10c6a9c4bad46deb63190d64169ec11" member_type: VOTER last_known_addr { host: "127.29.124.190" port: 35761 } }
I20240501 14:05:04.639926 4889 sys_catalog.cc:564] T 00000000000000000000000000000000 P f10c6a9c4bad46deb63190d64169ec11 [sys.catalog]: configured and running, proceeding with master startup.
I20240501 14:05:04.641932 4896 sys_catalog.cc:455] T 00000000000000000000000000000000 P f10c6a9c4bad46deb63190d64169ec11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f10c6a9c4bad46deb63190d64169ec11. Latest consensus state: current_term: 1 leader_uuid: "f10c6a9c4bad46deb63190d64169ec11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f10c6a9c4bad46deb63190d64169ec11" member_type: VOTER last_known_addr { host: "127.29.124.190" port: 35761 } } }
I20240501 14:05:04.642282 4896 sys_catalog.cc:458] T 00000000000000000000000000000000 P f10c6a9c4bad46deb63190d64169ec11 [sys.catalog]: This master's current role is: LEADER
I20240501 14:05:04.642763 4895 sys_catalog.cc:455] T 00000000000000000000000000000000 P f10c6a9c4bad46deb63190d64169ec11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f10c6a9c4bad46deb63190d64169ec11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f10c6a9c4bad46deb63190d64169ec11" member_type: VOTER last_known_addr { host: "127.29.124.190" port: 35761 } } }
I20240501 14:05:04.643083 4895 sys_catalog.cc:458] T 00000000000000000000000000000000 P f10c6a9c4bad46deb63190d64169ec11 [sys.catalog]: This master's current role is: LEADER
I20240501 14:05:04.644279 4902 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240501 14:05:04.649691 4902 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240501 14:05:04.659875 4902 catalog_manager.cc:1347] Generated new cluster ID: 05bb0ebbddb2424f888ce122a8f22cd0
I20240501 14:05:04.659984 4902 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240501 14:05:04.690155 4902 catalog_manager.cc:1370] Generated new certificate authority record
I20240501 14:05:04.691385 4902 catalog_manager.cc:1504] Loading token signing keys...
I20240501 14:05:04.722676 4902 catalog_manager.cc:5857] T 00000000000000000000000000000000 P f10c6a9c4bad46deb63190d64169ec11: Generated new TSK 0
I20240501 14:05:04.723898 4902 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240501 14:05:04.739715 30194 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUv7aek/build/asan/bin/kudu
/tmp/dist-test-taskUv7aek/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.124.129:0
--local_ip_for_outbound_sockets=127.29.124.129
--webserver_interface=127.29.124.129
--webserver_port=0
--tserver_master_addrs=127.29.124.190:35761
--builtin_ntp_servers=127.29.124.148:46431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 14:05:05.096361 4913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:05:05.096642 4913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:05:05.096726 4913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:05:05.104096 4913 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:05:05.104220 4913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:05:05.104329 4913 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.124.129
I20240501 14:05:05.113209 4913 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.124.148:46431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.124.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.29.124.129
--webserver_port=0
--tserver_master_addrs=127.29.124.190:35761
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.124.129
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:05:05.114603 4913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:05:05.116284 4913 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:05:05.128901 4919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:05:05.129349 4918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:05:05.129729 4921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:05:05.130182 4913 server_base.cc:1034] running on GCE node
I20240501 14:05:05.130746 4913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:05:05.131713 4913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:05:05.132915 4913 hybrid_clock.cc:648] HybridClock initialized: now 1714572305132847 us; error 57 us; skew 500 ppm
I20240501 14:05:05.133229 4913 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:05:05.135480 4913 webserver.cc:460] Webserver started at http://127.29.124.129:45397/ using document root <none> and password file <none>
I20240501 14:05:05.136185 4913 fs_manager.cc:359] Metadata directory not provided
I20240501 14:05:05.136282 4913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:05:05.136556 4913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:05:05.139189 4913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "04088ee3bea440a7bd5ff772152b9410"
format_stamp: "Formatted at 2024-05-01 14:05:05 on dist-test-slave-p83k"
I20240501 14:05:05.139884 4913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "04088ee3bea440a7bd5ff772152b9410"
format_stamp: "Formatted at 2024-05-01 14:05:05 on dist-test-slave-p83k"
I20240501 14:05:05.145525 4913 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240501 14:05:05.148861 4927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:05:05.150199 4913 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:05:05.150381 4913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "04088ee3bea440a7bd5ff772152b9410"
format_stamp: "Formatted at 2024-05-01 14:05:05 on dist-test-slave-p83k"
I20240501 14:05:05.150578 4913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:05:05.174736 4913 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:05:05.175635 4913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:05:05.175925 4913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:05:05.176877 4913 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:05:05.178419 4913 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:05:05.178519 4913 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:05:05.178627 4913 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:05:05.178692 4913 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:05:05.206869 4913 rpc_server.cc:307] RPC server started. Bound to: 127.29.124.129:44033
I20240501 14:05:05.206916 5039 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.124.129:44033 every 8 connection(s)
I20240501 14:05:05.208256 4913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240501 14:05:05.215571 30194 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUv7aek/build/asan/bin/kudu as pid 4913
I20240501 14:05:05.215775 30194 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240501 14:05:05.219345 30194 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUv7aek/build/asan/bin/kudu
/tmp/dist-test-taskUv7aek/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.124.130:0
--local_ip_for_outbound_sockets=127.29.124.130
--webserver_interface=127.29.124.130
--webserver_port=0
--tserver_master_addrs=127.29.124.190:35761
--builtin_ntp_servers=127.29.124.148:46431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:05:05.221482 5040 heartbeater.cc:347] Connected to a master server at 127.29.124.190:35761
I20240501 14:05:05.221907 5040 heartbeater.cc:464] Registering TS with master...
I20240501 14:05:05.222870 5040 heartbeater.cc:511] Master 127.29.124.190:35761 requested a full tablet report, sending...
I20240501 14:05:05.224988 4854 ts_manager.cc:194] Registered new tserver with Master: 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129:44033)
I20240501 14:05:05.226547 4854 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.124.129:44497
W20240501 14:05:05.545018 5044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:05:05.545334 5044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:05:05.545441 5044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:05:05.553004 5044 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:05:05.553139 5044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:05:05.553282 5044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.124.130
I20240501 14:05:05.562466 5044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.124.148:46431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.124.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.29.124.130
--webserver_port=0
--tserver_master_addrs=127.29.124.190:35761
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.124.130
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:05:05.563776 5044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:05:05.565567 5044 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:05:05.577764 5049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:05:05.577895 5052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:05:05.578284 5050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:05:05.578805 5044 server_base.cc:1034] running on GCE node
I20240501 14:05:05.579422 5044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:05:05.580404 5044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:05:05.581599 5044 hybrid_clock.cc:648] HybridClock initialized: now 1714572305581547 us; error 48 us; skew 500 ppm
I20240501 14:05:05.581930 5044 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:05:05.584084 5044 webserver.cc:460] Webserver started at http://127.29.124.130:34639/ using document root <none> and password file <none>
I20240501 14:05:05.584826 5044 fs_manager.cc:359] Metadata directory not provided
I20240501 14:05:05.584945 5044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:05:05.585275 5044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:05:05.587888 5044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "e9c60449cf3c46f8aca4ab6c535302a4"
format_stamp: "Formatted at 2024-05-01 14:05:05 on dist-test-slave-p83k"
I20240501 14:05:05.588609 5044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "e9c60449cf3c46f8aca4ab6c535302a4"
format_stamp: "Formatted at 2024-05-01 14:05:05 on dist-test-slave-p83k"
I20240501 14:05:05.593679 5044 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240501 14:05:05.597103 5058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:05:05.598419 5044 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:05:05.598574 5044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "e9c60449cf3c46f8aca4ab6c535302a4"
format_stamp: "Formatted at 2024-05-01 14:05:05 on dist-test-slave-p83k"
I20240501 14:05:05.598762 5044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:05:05.618961 5044 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:05:05.619817 5044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:05:05.620082 5044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:05:05.621021 5044 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:05:05.622609 5044 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:05:05.622709 5044 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:05:05.622817 5044 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:05:05.622884 5044 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:05:05.651171 5044 rpc_server.cc:307] RPC server started. Bound to: 127.29.124.130:34377
I20240501 14:05:05.651255 5170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.124.130:34377 every 8 connection(s)
I20240501 14:05:05.652484 5044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240501 14:05:05.657712 30194 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUv7aek/build/asan/bin/kudu as pid 5044
I20240501 14:05:05.657860 30194 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240501 14:05:05.660905 30194 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUv7aek/build/asan/bin/kudu
/tmp/dist-test-taskUv7aek/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.124.131:0
--local_ip_for_outbound_sockets=127.29.124.131
--webserver_interface=127.29.124.131
--webserver_port=0
--tserver_master_addrs=127.29.124.190:35761
--builtin_ntp_servers=127.29.124.148:46431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:05:05.665212 5171 heartbeater.cc:347] Connected to a master server at 127.29.124.190:35761
I20240501 14:05:05.665539 5171 heartbeater.cc:464] Registering TS with master...
I20240501 14:05:05.666363 5171 heartbeater.cc:511] Master 127.29.124.190:35761 requested a full tablet report, sending...
I20240501 14:05:05.667677 4854 ts_manager.cc:194] Registered new tserver with Master: e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377)
I20240501 14:05:05.668422 4854 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.124.130:48005
W20240501 14:05:05.988550 5174 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:05:05.988849 5174 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:05:05.988978 5174 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:05:05.996500 5174 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:05:05.996629 5174 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:05:05.996779 5174 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.124.131
I20240501 14:05:06.005832 5174 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.124.148:46431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.124.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.29.124.131
--webserver_port=0
--tserver_master_addrs=127.29.124.190:35761
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.124.131
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:05:06.007179 5174 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:05:06.008924 5174 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:05:06.022246 5180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:05:06.022516 5182 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:05:06.023015 5179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:05:06.023626 5174 server_base.cc:1034] running on GCE node
I20240501 14:05:06.024199 5174 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:05:06.025115 5174 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:05:06.026319 5174 hybrid_clock.cc:648] HybridClock initialized: now 1714572306026276 us; error 56 us; skew 500 ppm
I20240501 14:05:06.026666 5174 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:05:06.028853 5174 webserver.cc:460] Webserver started at http://127.29.124.131:45349/ using document root <none> and password file <none>
I20240501 14:05:06.029635 5174 fs_manager.cc:359] Metadata directory not provided
I20240501 14:05:06.029742 5174 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:05:06.030061 5174 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:05:06.032653 5174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "1c3fdc5272484ac09ef3e5e36a08f27c"
format_stamp: "Formatted at 2024-05-01 14:05:06 on dist-test-slave-p83k"
I20240501 14:05:06.033438 5174 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "1c3fdc5272484ac09ef3e5e36a08f27c"
format_stamp: "Formatted at 2024-05-01 14:05:06 on dist-test-slave-p83k"
I20240501 14:05:06.038615 5174 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240501 14:05:06.041994 5188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:05:06.043406 5174 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 14:05:06.043622 5174 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "1c3fdc5272484ac09ef3e5e36a08f27c"
format_stamp: "Formatted at 2024-05-01 14:05:06 on dist-test-slave-p83k"
I20240501 14:05:06.043846 5174 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:05:06.066063 5174 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:05:06.066967 5174 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:05:06.067246 5174 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:05:06.068229 5174 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:05:06.069778 5174 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:05:06.069872 5174 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:05:06.069985 5174 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:05:06.070050 5174 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:05:06.099743 5174 rpc_server.cc:307] RPC server started. Bound to: 127.29.124.131:33209
I20240501 14:05:06.099790 5300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.124.131:33209 every 8 connection(s)
I20240501 14:05:06.101052 5174 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240501 14:05:06.109100 30194 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUv7aek/build/asan/bin/kudu as pid 5174
I20240501 14:05:06.109290 30194 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240501 14:05:06.112795 30194 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUv7aek/build/asan/bin/kudu
/tmp/dist-test-taskUv7aek/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.124.132:0
--local_ip_for_outbound_sockets=127.29.124.132
--webserver_interface=127.29.124.132
--webserver_port=0
--tserver_master_addrs=127.29.124.190:35761
--builtin_ntp_servers=127.29.124.148:46431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:05:06.113816 5301 heartbeater.cc:347] Connected to a master server at 127.29.124.190:35761
I20240501 14:05:06.114178 5301 heartbeater.cc:464] Registering TS with master...
I20240501 14:05:06.115213 5301 heartbeater.cc:511] Master 127.29.124.190:35761 requested a full tablet report, sending...
I20240501 14:05:06.116801 4854 ts_manager.cc:194] Registered new tserver with Master: 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209)
I20240501 14:05:06.117722 4854 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.124.131:60463
I20240501 14:05:06.229753 5040 heartbeater.cc:503] Master 127.29.124.190:35761 was elected leader, sending a full tablet report...
W20240501 14:05:06.444089 5304 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:05:06.444396 5304 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:05:06.444525 5304 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:05:06.452028 5304 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:05:06.452172 5304 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:05:06.452328 5304 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.124.132
I20240501 14:05:06.461344 5304 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.124.148:46431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.124.132:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.29.124.132
--webserver_port=0
--tserver_master_addrs=127.29.124.190:35761
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.124.132
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:05:06.462644 5304 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:05:06.464329 5304 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:05:06.476176 5310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:05:06.476401 5312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:05:06.476867 5309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:05:06.479080 5304 server_base.cc:1034] running on GCE node
I20240501 14:05:06.479709 5304 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:05:06.480633 5304 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:05:06.481834 5304 hybrid_clock.cc:648] HybridClock initialized: now 1714572306481784 us; error 44 us; skew 500 ppm
I20240501 14:05:06.482156 5304 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:05:06.484297 5304 webserver.cc:460] Webserver started at http://127.29.124.132:44477/ using document root <none> and password file <none>
I20240501 14:05:06.484998 5304 fs_manager.cc:359] Metadata directory not provided
I20240501 14:05:06.485116 5304 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:05:06.485590 5304 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:05:06.488178 5304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "be56773105e744919601127a94e98adf"
format_stamp: "Formatted at 2024-05-01 14:05:06 on dist-test-slave-p83k"
I20240501 14:05:06.488909 5304 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "be56773105e744919601127a94e98adf"
format_stamp: "Formatted at 2024-05-01 14:05:06 on dist-test-slave-p83k"
I20240501 14:05:06.493991 5304 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20240501 14:05:06.497231 5318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:05:06.498579 5304 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 14:05:06.498733 5304 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "be56773105e744919601127a94e98adf"
format_stamp: "Formatted at 2024-05-01 14:05:06 on dist-test-slave-p83k"
I20240501 14:05:06.498916 5304 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:05:06.513659 5304 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:05:06.514458 5304 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:05:06.514705 5304 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:05:06.515578 5304 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:05:06.516971 5304 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:05:06.517061 5304 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:05:06.517179 5304 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:05:06.517241 5304 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:05:06.544512 5304 rpc_server.cc:307] RPC server started. Bound to: 127.29.124.132:35459
I20240501 14:05:06.544576 5430 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.124.132:35459 every 8 connection(s)
I20240501 14:05:06.545845 5304 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240501 14:05:06.551580 30194 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUv7aek/build/asan/bin/kudu as pid 5304
I20240501 14:05:06.551776 30194 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240501 14:05:06.555171 30194 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUv7aek/build/asan/bin/kudu
/tmp/dist-test-taskUv7aek/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.124.133:0
--local_ip_for_outbound_sockets=127.29.124.133
--webserver_interface=127.29.124.133
--webserver_port=0
--tserver_master_addrs=127.29.124.190:35761
--builtin_ntp_servers=127.29.124.148:46431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:05:06.559558 5431 heartbeater.cc:347] Connected to a master server at 127.29.124.190:35761
I20240501 14:05:06.559854 5431 heartbeater.cc:464] Registering TS with master...
I20240501 14:05:06.560678 5431 heartbeater.cc:511] Master 127.29.124.190:35761 requested a full tablet report, sending...
I20240501 14:05:06.562043 4854 ts_manager.cc:194] Registered new tserver with Master: be56773105e744919601127a94e98adf (127.29.124.132:35459)
I20240501 14:05:06.562808 4854 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.124.132:51497
I20240501 14:05:06.670866 5171 heartbeater.cc:503] Master 127.29.124.190:35761 was elected leader, sending a full tablet report...
W20240501 14:05:06.887912 5434 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:05:06.888211 5434 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:05:06.888326 5434 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:05:06.895785 5434 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:05:06.895910 5434 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:05:06.896061 5434 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.124.133
I20240501 14:05:06.905169 5434 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.124.148:46431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.124.133:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.29.124.133
--webserver_port=0
--tserver_master_addrs=127.29.124.190:35761
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.124.133
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:05:06.906546 5434 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:05:06.908265 5434 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:05:06.920181 5439 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:05:06.920420 5442 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:05:06.920910 5440 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:05:06.921567 5434 server_base.cc:1034] running on GCE node
I20240501 14:05:06.922135 5434 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:05:06.923046 5434 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:05:06.924274 5434 hybrid_clock.cc:648] HybridClock initialized: now 1714572306924223 us; error 39 us; skew 500 ppm
I20240501 14:05:06.924597 5434 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:05:06.926848 5434 webserver.cc:460] Webserver started at http://127.29.124.133:46525/ using document root <none> and password file <none>
I20240501 14:05:06.927565 5434 fs_manager.cc:359] Metadata directory not provided
I20240501 14:05:06.927671 5434 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:05:06.927976 5434 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:05:06.930569 5434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "938a24b0e6a040b7966225af26558ff8"
format_stamp: "Formatted at 2024-05-01 14:05:06 on dist-test-slave-p83k"
I20240501 14:05:06.931286 5434 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "938a24b0e6a040b7966225af26558ff8"
format_stamp: "Formatted at 2024-05-01 14:05:06 on dist-test-slave-p83k"
I20240501 14:05:06.936271 5434 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20240501 14:05:06.939610 5448 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:05:06.940886 5434 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20240501 14:05:06.941040 5434 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "938a24b0e6a040b7966225af26558ff8"
format_stamp: "Formatted at 2024-05-01 14:05:06 on dist-test-slave-p83k"
I20240501 14:05:06.941221 5434 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:05:06.956274 5434 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:05:06.957088 5434 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:05:06.957388 5434 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:05:06.958302 5434 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:05:06.959715 5434 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:05:06.959800 5434 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:05:06.959908 5434 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:05:06.959970 5434 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:05:06.989567 5434 rpc_server.cc:307] RPC server started. Bound to: 127.29.124.133:35635
I20240501 14:05:06.989620 5560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.124.133:35635 every 8 connection(s)
I20240501 14:05:06.990895 5434 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 14:05:06.994827 30194 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUv7aek/build/asan/bin/kudu as pid 5434
I20240501 14:05:06.995024 30194 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240501 14:05:06.998442 30194 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUv7aek/build/asan/bin/kudu
/tmp/dist-test-taskUv7aek/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.124.134:0
--local_ip_for_outbound_sockets=127.29.124.134
--webserver_interface=127.29.124.134
--webserver_port=0
--tserver_master_addrs=127.29.124.190:35761
--builtin_ntp_servers=127.29.124.148:46431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:05:07.004168 5561 heartbeater.cc:347] Connected to a master server at 127.29.124.190:35761
I20240501 14:05:07.004482 5561 heartbeater.cc:464] Registering TS with master...
I20240501 14:05:07.005373 5561 heartbeater.cc:511] Master 127.29.124.190:35761 requested a full tablet report, sending...
I20240501 14:05:07.006867 4854 ts_manager.cc:194] Registered new tserver with Master: 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:05:07.007922 4854 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.124.133:48537
I20240501 14:05:07.120636 5301 heartbeater.cc:503] Master 127.29.124.190:35761 was elected leader, sending a full tablet report...
W20240501 14:05:07.328006 5564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:05:07.328299 5564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:05:07.328423 5564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:05:07.335740 5564 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:05:07.335877 5564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:05:07.336028 5564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.124.134
I20240501 14:05:07.345032 5564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.124.148:46431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.124.134:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.29.124.134
--webserver_port=0
--tserver_master_addrs=127.29.124.190:35761
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.124.134
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:05:07.346387 5564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:05:07.348204 5564 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:05:07.360087 5569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:05:07.360320 5570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:05:07.360797 5572 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:05:07.361373 5564 server_base.cc:1034] running on GCE node
I20240501 14:05:07.361940 5564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:05:07.362854 5564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:05:07.364056 5564 hybrid_clock.cc:648] HybridClock initialized: now 1714572307364007 us; error 37 us; skew 500 ppm
I20240501 14:05:07.364380 5564 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:05:07.373661 5564 webserver.cc:460] Webserver started at http://127.29.124.134:37279/ using document root <none> and password file <none>
I20240501 14:05:07.374471 5564 fs_manager.cc:359] Metadata directory not provided
I20240501 14:05:07.374591 5564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:05:07.374945 5564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:05:07.377944 5564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "19a6d2d8a08142d5bffcde3a37234692"
format_stamp: "Formatted at 2024-05-01 14:05:07 on dist-test-slave-p83k"
I20240501 14:05:07.378692 5564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "19a6d2d8a08142d5bffcde3a37234692"
format_stamp: "Formatted at 2024-05-01 14:05:07 on dist-test-slave-p83k"
I20240501 14:05:07.383910 5564 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240501 14:05:07.387291 5578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:05:07.388660 5564 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 14:05:07.388819 5564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "19a6d2d8a08142d5bffcde3a37234692"
format_stamp: "Formatted at 2024-05-01 14:05:07 on dist-test-slave-p83k"
I20240501 14:05:07.389006 5564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:05:07.414418 5564 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:05:07.415264 5564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:05:07.415518 5564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:05:07.416456 5564 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:05:07.417922 5564 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:05:07.418010 5564 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:05:07.418100 5564 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:05:07.418154 5564 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:05:07.445094 5564 rpc_server.cc:307] RPC server started. Bound to: 127.29.124.134:40665
I20240501 14:05:07.445165 5690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.124.134:40665 every 8 connection(s)
I20240501 14:05:07.446473 5564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 14:05:07.449065 30194 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUv7aek/build/asan/bin/kudu as pid 5564
I20240501 14:05:07.449242 30194 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240501 14:05:07.452590 30194 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUv7aek/build/asan/bin/kudu
/tmp/dist-test-taskUv7aek/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.124.135:0
--local_ip_for_outbound_sockets=127.29.124.135
--webserver_interface=127.29.124.135
--webserver_port=0
--tserver_master_addrs=127.29.124.190:35761
--builtin_ntp_servers=127.29.124.148:46431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:05:07.459617 5691 heartbeater.cc:347] Connected to a master server at 127.29.124.190:35761
I20240501 14:05:07.459910 5691 heartbeater.cc:464] Registering TS with master...
I20240501 14:05:07.460700 5691 heartbeater.cc:511] Master 127.29.124.190:35761 requested a full tablet report, sending...
I20240501 14:05:07.462083 4854 ts_manager.cc:194] Registered new tserver with Master: 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665)
I20240501 14:05:07.462914 4854 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.124.134:35771
I20240501 14:05:07.565202 5431 heartbeater.cc:503] Master 127.29.124.190:35761 was elected leader, sending a full tablet report...
W20240501 14:05:07.798247 5694 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:05:07.798537 5694 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:05:07.798661 5694 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:05:07.806263 5694 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:05:07.806386 5694 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:05:07.806531 5694 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.124.135
I20240501 14:05:07.815524 5694 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.124.148:46431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.124.135:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.29.124.135
--webserver_port=0
--tserver_master_addrs=127.29.124.190:35761
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.124.135
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:05:07.816838 5694 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:05:07.818570 5694 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:05:07.830473 5699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:05:07.830817 5702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:05:07.831308 5700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:05:07.831883 5694 server_base.cc:1034] running on GCE node
I20240501 14:05:07.832475 5694 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:05:07.833448 5694 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:05:07.834658 5694 hybrid_clock.cc:648] HybridClock initialized: now 1714572307834607 us; error 43 us; skew 500 ppm
I20240501 14:05:07.834980 5694 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:05:07.837096 5694 webserver.cc:460] Webserver started at http://127.29.124.135:37713/ using document root <none> and password file <none>
I20240501 14:05:07.837858 5694 fs_manager.cc:359] Metadata directory not provided
I20240501 14:05:07.837971 5694 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:05:07.838287 5694 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:05:07.840991 5694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "4360b59e3a154688a79ca5f9c0a37bc9"
format_stamp: "Formatted at 2024-05-01 14:05:07 on dist-test-slave-p83k"
I20240501 14:05:07.841742 5694 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "4360b59e3a154688a79ca5f9c0a37bc9"
format_stamp: "Formatted at 2024-05-01 14:05:07 on dist-test-slave-p83k"
I20240501 14:05:07.846904 5694 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240501 14:05:07.850443 5708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:05:07.851763 5694 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:05:07.851902 5694 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "4360b59e3a154688a79ca5f9c0a37bc9"
format_stamp: "Formatted at 2024-05-01 14:05:07 on dist-test-slave-p83k"
I20240501 14:05:07.852079 5694 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:05:07.870416 5694 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:05:07.871222 5694 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:05:07.871469 5694 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:05:07.872375 5694 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:05:07.873803 5694 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:05:07.873888 5694 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:05:07.874008 5694 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:05:07.874070 5694 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:05:07.902087 5694 rpc_server.cc:307] RPC server started. Bound to: 127.29.124.135:35329
I20240501 14:05:07.902146 5820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.124.135:35329 every 8 connection(s)
I20240501 14:05:07.903522 5694 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 14:05:07.912770 30194 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUv7aek/build/asan/bin/kudu as pid 5694
I20240501 14:05:07.912955 30194 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240501 14:05:07.915581 5821 heartbeater.cc:347] Connected to a master server at 127.29.124.190:35761
I20240501 14:05:07.915858 5821 heartbeater.cc:464] Registering TS with master...
I20240501 14:05:07.916205 30194 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUv7aek/build/asan/bin/kudu
/tmp/dist-test-taskUv7aek/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.124.136:0
--local_ip_for_outbound_sockets=127.29.124.136
--webserver_interface=127.29.124.136
--webserver_port=0
--tserver_master_addrs=127.29.124.190:35761
--builtin_ntp_servers=127.29.124.148:46431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:05:07.916682 5821 heartbeater.cc:511] Master 127.29.124.190:35761 requested a full tablet report, sending...
I20240501 14:05:07.918049 4854 ts_manager.cc:194] Registered new tserver with Master: 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329)
I20240501 14:05:07.919004 4854 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.124.135:47951
I20240501 14:05:08.010870 5561 heartbeater.cc:503] Master 127.29.124.190:35761 was elected leader, sending a full tablet report...
W20240501 14:05:08.251403 5824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:05:08.251693 5824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:05:08.251809 5824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:05:08.259354 5824 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:05:08.259478 5824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:05:08.259631 5824 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.124.136
I20240501 14:05:08.268509 5824 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.124.148:46431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.124.136:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.29.124.136
--webserver_port=0
--tserver_master_addrs=127.29.124.190:35761
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.124.136
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:05:08.269848 5824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:05:08.271569 5824 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:05:08.283279 5829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:05:08.283679 5830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:05:08.284121 5832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:05:08.284739 5824 server_base.cc:1034] running on GCE node
I20240501 14:05:08.285377 5824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:05:08.286366 5824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:05:08.287580 5824 hybrid_clock.cc:648] HybridClock initialized: now 1714572308287519 us; error 53 us; skew 500 ppm
I20240501 14:05:08.287897 5824 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:05:08.293141 5824 webserver.cc:460] Webserver started at http://127.29.124.136:37805/ using document root <none> and password file <none>
I20240501 14:05:08.293998 5824 fs_manager.cc:359] Metadata directory not provided
I20240501 14:05:08.294116 5824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:05:08.294456 5824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240501 14:05:08.297084 5824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "6dac90f01e744f9c8701a404440a4c77"
format_stamp: "Formatted at 2024-05-01 14:05:08 on dist-test-slave-p83k"
I20240501 14:05:08.297861 5824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "6dac90f01e744f9c8701a404440a4c77"
format_stamp: "Formatted at 2024-05-01 14:05:08 on dist-test-slave-p83k"
I20240501 14:05:08.303085 5824 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20240501 14:05:08.306531 5838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:05:08.307823 5824 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240501 14:05:08.307982 5824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "6dac90f01e744f9c8701a404440a4c77"
format_stamp: "Formatted at 2024-05-01 14:05:08 on dist-test-slave-p83k"
I20240501 14:05:08.308187 5824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:05:08.321960 5824 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:05:08.322798 5824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:05:08.323060 5824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:05:08.323990 5824 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:05:08.325479 5824 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:05:08.325573 5824 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:05:08.325676 5824 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:05:08.325742 5824 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:05:08.354485 5824 rpc_server.cc:307] RPC server started. Bound to: 127.29.124.136:43377
I20240501 14:05:08.354532 5950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.124.136:43377 every 8 connection(s)
I20240501 14:05:08.355840 5824 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 14:05:08.364171 30194 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUv7aek/build/asan/bin/kudu as pid 5824
I20240501 14:05:08.364328 30194 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240501 14:05:08.367923 5951 heartbeater.cc:347] Connected to a master server at 127.29.124.190:35761
I20240501 14:05:08.368202 5951 heartbeater.cc:464] Registering TS with master...
I20240501 14:05:08.368970 5951 heartbeater.cc:511] Master 127.29.124.190:35761 requested a full tablet report, sending...
I20240501 14:05:08.370297 4853 ts_manager.cc:194] Registered new tserver with Master: 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377)
I20240501 14:05:08.371170 4853 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.124.136:49133
I20240501 14:05:08.380529 30194 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 14:05:08.396677 30194 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240501 14:05:08.396793 30194 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUv7aek/build/asan/bin/kudu pid 5434
I20240501 14:05:08.404647 5556 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:05:08.465731 5691 heartbeater.cc:503] Master 127.29.124.190:35761 was elected leader, sending a full tablet report...
I20240501 14:05:08.484737 30194 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUv7aek/build/asan/bin/kudu with pid 5434
I20240501 14:05:08.497838 30194 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUv7aek/build/asan/bin/kudu pid 5564
I20240501 14:05:08.505875 5686 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:05:08.585743 30194 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUv7aek/build/asan/bin/kudu with pid 5564
I20240501 14:05:08.628182 30194 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUv7aek/build/asan/bin/kudu pid 5694
I20240501 14:05:08.635756 5816 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:05:08.715178 30194 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUv7aek/build/asan/bin/kudu with pid 5694
I20240501 14:05:08.728459 30194 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskUv7aek/build/asan/bin/kudu pid 5824
I20240501 14:05:08.736471 5946 generic_service.cc:196] Checking for leaks (request via RPC)
I20240501 14:05:08.820470 30194 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskUv7aek/build/asan/bin/kudu with pid 5824
I20240501 14:05:12.585419 4853 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55620:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:05:12.649224 4974 tablet_service.cc:1467] Processing CreateTablet for tablet 88ed5438fc934adc9fb98c6cb4476598 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7af67fe405db4bffbc53cb288b8bb715]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:12.650774 4974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88ed5438fc934adc9fb98c6cb4476598. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:12.651322 5236 tablet_service.cc:1467] Processing CreateTablet for tablet 9b88c43cb8704583a3e65c5c35c205f6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7af67fe405db4bffbc53cb288b8bb715]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:12.651685 5105 tablet_service.cc:1467] Processing CreateTablet for tablet 43b46fcf827c4275a47291afb107c290 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7af67fe405db4bffbc53cb288b8bb715]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:12.652657 5236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b88c43cb8704583a3e65c5c35c205f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:12.652704 5105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43b46fcf827c4275a47291afb107c290. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:12.649209 4973 tablet_service.cc:1467] Processing CreateTablet for tablet 43b46fcf827c4275a47291afb107c290 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7af67fe405db4bffbc53cb288b8bb715]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:12.655304 4973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43b46fcf827c4275a47291afb107c290. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:12.651448 5235 tablet_service.cc:1467] Processing CreateTablet for tablet 88ed5438fc934adc9fb98c6cb4476598 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7af67fe405db4bffbc53cb288b8bb715]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:12.657784 5235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88ed5438fc934adc9fb98c6cb4476598. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:12.661413 4975 tablet_service.cc:1467] Processing CreateTablet for tablet 9b88c43cb8704583a3e65c5c35c205f6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7af67fe405db4bffbc53cb288b8bb715]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:12.651325 5106 tablet_service.cc:1467] Processing CreateTablet for tablet 88ed5438fc934adc9fb98c6cb4476598 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7af67fe405db4bffbc53cb288b8bb715]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:12.662040 5106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88ed5438fc934adc9fb98c6cb4476598. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:12.662499 4975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b88c43cb8704583a3e65c5c35c205f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:12.672511 5981 tablet_bootstrap.cc:492] T 88ed5438fc934adc9fb98c6cb4476598 P 1c3fdc5272484ac09ef3e5e36a08f27c: Bootstrap starting.
I20240501 14:05:12.678393 5366 tablet_service.cc:1467] Processing CreateTablet for tablet 9b88c43cb8704583a3e65c5c35c205f6 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7af67fe405db4bffbc53cb288b8bb715]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:12.679098 5365 tablet_service.cc:1467] Processing CreateTablet for tablet 43b46fcf827c4275a47291afb107c290 (DEFAULT_TABLE table=rebalance_test_table_0 [id=7af67fe405db4bffbc53cb288b8bb715]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:12.679723 5980 tablet_bootstrap.cc:492] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4: Bootstrap starting.
I20240501 14:05:12.680179 5365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43b46fcf827c4275a47291afb107c290. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:12.684257 5366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b88c43cb8704583a3e65c5c35c205f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:12.684376 5980 tablet_bootstrap.cc:654] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:12.685381 5981 tablet_bootstrap.cc:654] T 88ed5438fc934adc9fb98c6cb4476598 P 1c3fdc5272484ac09ef3e5e36a08f27c: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:12.685727 5980 log.cc:824] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4: Log is configured to *not* fsync() on all Append() calls
I20240501 14:05:12.686970 5981 log.cc:824] T 88ed5438fc934adc9fb98c6cb4476598 P 1c3fdc5272484ac09ef3e5e36a08f27c: Log is configured to *not* fsync() on all Append() calls
I20240501 14:05:12.687165 5982 tablet_bootstrap.cc:492] T 88ed5438fc934adc9fb98c6cb4476598 P 04088ee3bea440a7bd5ff772152b9410: Bootstrap starting.
I20240501 14:05:12.690920 5982 tablet_bootstrap.cc:654] T 88ed5438fc934adc9fb98c6cb4476598 P 04088ee3bea440a7bd5ff772152b9410: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:12.692261 5982 log.cc:824] T 88ed5438fc934adc9fb98c6cb4476598 P 04088ee3bea440a7bd5ff772152b9410: Log is configured to *not* fsync() on all Append() calls
I20240501 14:05:12.701876 5982 tablet_bootstrap.cc:492] T 88ed5438fc934adc9fb98c6cb4476598 P 04088ee3bea440a7bd5ff772152b9410: No bootstrap required, opened a new log
I20240501 14:05:12.702306 5982 ts_tablet_manager.cc:1367] T 88ed5438fc934adc9fb98c6cb4476598 P 04088ee3bea440a7bd5ff772152b9410: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.007s
I20240501 14:05:12.703963 5986 tablet_bootstrap.cc:492] T 43b46fcf827c4275a47291afb107c290 P be56773105e744919601127a94e98adf: Bootstrap starting.
I20240501 14:05:12.708237 5980 tablet_bootstrap.cc:492] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4: No bootstrap required, opened a new log
I20240501 14:05:12.708544 5980 ts_tablet_manager.cc:1367] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.019s
I20240501 14:05:12.708722 5986 tablet_bootstrap.cc:654] T 43b46fcf827c4275a47291afb107c290 P be56773105e744919601127a94e98adf: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:12.710181 5986 log.cc:824] T 43b46fcf827c4275a47291afb107c290 P be56773105e744919601127a94e98adf: Log is configured to *not* fsync() on all Append() calls
I20240501 14:05:12.711705 5982 raft_consensus.cc:348] T 88ed5438fc934adc9fb98c6cb4476598 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } }
I20240501 14:05:12.712294 5982 raft_consensus.cc:374] T 88ed5438fc934adc9fb98c6cb4476598 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:12.712438 5982 raft_consensus.cc:729] T 88ed5438fc934adc9fb98c6cb4476598 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04088ee3bea440a7bd5ff772152b9410, State: Initialized, Role: FOLLOWER
I20240501 14:05:12.713325 5982 consensus_queue.cc:260] T 88ed5438fc934adc9fb98c6cb4476598 P 04088ee3bea440a7bd5ff772152b9410 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } }
I20240501 14:05:12.717223 5980 raft_consensus.cc:348] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } }
I20240501 14:05:12.718439 5982 ts_tablet_manager.cc:1398] T 88ed5438fc934adc9fb98c6cb4476598 P 04088ee3bea440a7bd5ff772152b9410: Time spent starting tablet: real 0.016s user 0.014s sys 0.001s
I20240501 14:05:12.718423 5980 raft_consensus.cc:374] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:12.718533 5980 raft_consensus.cc:729] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9c60449cf3c46f8aca4ab6c535302a4, State: Initialized, Role: FOLLOWER
I20240501 14:05:12.719102 5986 tablet_bootstrap.cc:492] T 43b46fcf827c4275a47291afb107c290 P be56773105e744919601127a94e98adf: No bootstrap required, opened a new log
I20240501 14:05:12.719216 5980 consensus_queue.cc:260] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } }
I20240501 14:05:12.719447 5986 ts_tablet_manager.cc:1367] T 43b46fcf827c4275a47291afb107c290 P be56773105e744919601127a94e98adf: Time spent bootstrapping tablet: real 0.016s user 0.014s sys 0.000s
I20240501 14:05:12.719580 5981 tablet_bootstrap.cc:492] T 88ed5438fc934adc9fb98c6cb4476598 P 1c3fdc5272484ac09ef3e5e36a08f27c: No bootstrap required, opened a new log
I20240501 14:05:12.719902 5981 ts_tablet_manager.cc:1367] T 88ed5438fc934adc9fb98c6cb4476598 P 1c3fdc5272484ac09ef3e5e36a08f27c: Time spent bootstrapping tablet: real 0.048s user 0.020s sys 0.016s
I20240501 14:05:12.720633 5980 ts_tablet_manager.cc:1398] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20240501 14:05:12.720976 5980 tablet_bootstrap.cc:492] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4: Bootstrap starting.
I20240501 14:05:12.722587 5980 tablet_bootstrap.cc:654] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:12.726477 5982 tablet_bootstrap.cc:492] T 43b46fcf827c4275a47291afb107c290 P 04088ee3bea440a7bd5ff772152b9410: Bootstrap starting.
I20240501 14:05:12.728145 5982 tablet_bootstrap.cc:654] T 43b46fcf827c4275a47291afb107c290 P 04088ee3bea440a7bd5ff772152b9410: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:12.730228 5981 raft_consensus.cc:348] T 88ed5438fc934adc9fb98c6cb4476598 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } }
I20240501 14:05:12.730769 5981 raft_consensus.cc:374] T 88ed5438fc934adc9fb98c6cb4476598 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:12.730906 5981 raft_consensus.cc:729] T 88ed5438fc934adc9fb98c6cb4476598 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c3fdc5272484ac09ef3e5e36a08f27c, State: Initialized, Role: FOLLOWER
I20240501 14:05:12.731719 5981 consensus_queue.cc:260] T 88ed5438fc934adc9fb98c6cb4476598 P 1c3fdc5272484ac09ef3e5e36a08f27c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } }
I20240501 14:05:12.733144 5982 tablet_bootstrap.cc:492] T 43b46fcf827c4275a47291afb107c290 P 04088ee3bea440a7bd5ff772152b9410: No bootstrap required, opened a new log
I20240501 14:05:12.733330 5982 ts_tablet_manager.cc:1367] T 43b46fcf827c4275a47291afb107c290 P 04088ee3bea440a7bd5ff772152b9410: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.004s
I20240501 14:05:12.733410 5986 raft_consensus.cc:348] T 43b46fcf827c4275a47291afb107c290 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } }
I20240501 14:05:12.733872 5986 raft_consensus.cc:374] T 43b46fcf827c4275a47291afb107c290 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:12.733992 5986 raft_consensus.cc:729] T 43b46fcf827c4275a47291afb107c290 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be56773105e744919601127a94e98adf, State: Initialized, Role: FOLLOWER
I20240501 14:05:12.734385 5982 raft_consensus.cc:348] T 43b46fcf827c4275a47291afb107c290 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } }
I20240501 14:05:12.734609 5982 raft_consensus.cc:374] T 43b46fcf827c4275a47291afb107c290 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:12.734702 5982 raft_consensus.cc:729] T 43b46fcf827c4275a47291afb107c290 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04088ee3bea440a7bd5ff772152b9410, State: Initialized, Role: FOLLOWER
I20240501 14:05:12.734781 5986 consensus_queue.cc:260] T 43b46fcf827c4275a47291afb107c290 P be56773105e744919601127a94e98adf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } }
I20240501 14:05:12.734937 5982 consensus_queue.cc:260] T 43b46fcf827c4275a47291afb107c290 P 04088ee3bea440a7bd5ff772152b9410 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } }
I20240501 14:05:12.735975 5982 ts_tablet_manager.cc:1398] T 43b46fcf827c4275a47291afb107c290 P 04088ee3bea440a7bd5ff772152b9410: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:05:12.736281 5982 tablet_bootstrap.cc:492] T 9b88c43cb8704583a3e65c5c35c205f6 P 04088ee3bea440a7bd5ff772152b9410: Bootstrap starting.
I20240501 14:05:12.738233 5982 tablet_bootstrap.cc:654] T 9b88c43cb8704583a3e65c5c35c205f6 P 04088ee3bea440a7bd5ff772152b9410: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:12.739827 5986 ts_tablet_manager.cc:1398] T 43b46fcf827c4275a47291afb107c290 P be56773105e744919601127a94e98adf: Time spent starting tablet: real 0.020s user 0.010s sys 0.004s
I20240501 14:05:12.739993 5980 tablet_bootstrap.cc:492] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4: No bootstrap required, opened a new log
I20240501 14:05:12.740219 5980 ts_tablet_manager.cc:1367] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.010s
I20240501 14:05:12.740420 5986 tablet_bootstrap.cc:492] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf: Bootstrap starting.
I20240501 14:05:12.741438 5980 raft_consensus.cc:348] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } }
I20240501 14:05:12.741714 5980 raft_consensus.cc:374] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:12.741834 5980 raft_consensus.cc:729] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9c60449cf3c46f8aca4ab6c535302a4, State: Initialized, Role: FOLLOWER
I20240501 14:05:12.742098 5980 consensus_queue.cc:260] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } }
I20240501 14:05:12.742237 5986 tablet_bootstrap.cc:654] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:12.742666 5980 ts_tablet_manager.cc:1398] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:05:12.743703 5981 ts_tablet_manager.cc:1398] T 88ed5438fc934adc9fb98c6cb4476598 P 1c3fdc5272484ac09ef3e5e36a08f27c: Time spent starting tablet: real 0.024s user 0.014s sys 0.003s
I20240501 14:05:12.744202 5981 tablet_bootstrap.cc:492] T 9b88c43cb8704583a3e65c5c35c205f6 P 1c3fdc5272484ac09ef3e5e36a08f27c: Bootstrap starting.
I20240501 14:05:12.745960 5981 tablet_bootstrap.cc:654] T 9b88c43cb8704583a3e65c5c35c205f6 P 1c3fdc5272484ac09ef3e5e36a08f27c: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:12.749444 5982 tablet_bootstrap.cc:492] T 9b88c43cb8704583a3e65c5c35c205f6 P 04088ee3bea440a7bd5ff772152b9410: No bootstrap required, opened a new log
I20240501 14:05:12.749692 5982 ts_tablet_manager.cc:1367] T 9b88c43cb8704583a3e65c5c35c205f6 P 04088ee3bea440a7bd5ff772152b9410: Time spent bootstrapping tablet: real 0.013s user 0.001s sys 0.011s
I20240501 14:05:12.750576 5982 raft_consensus.cc:348] T 9b88c43cb8704583a3e65c5c35c205f6 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:12.750902 5982 raft_consensus.cc:374] T 9b88c43cb8704583a3e65c5c35c205f6 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:12.751016 5982 raft_consensus.cc:729] T 9b88c43cb8704583a3e65c5c35c205f6 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04088ee3bea440a7bd5ff772152b9410, State: Initialized, Role: FOLLOWER
I20240501 14:05:12.751452 5982 consensus_queue.cc:260] T 9b88c43cb8704583a3e65c5c35c205f6 P 04088ee3bea440a7bd5ff772152b9410 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:12.752096 5982 ts_tablet_manager.cc:1398] T 9b88c43cb8704583a3e65c5c35c205f6 P 04088ee3bea440a7bd5ff772152b9410: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:05:12.753237 5981 tablet_bootstrap.cc:492] T 9b88c43cb8704583a3e65c5c35c205f6 P 1c3fdc5272484ac09ef3e5e36a08f27c: No bootstrap required, opened a new log
I20240501 14:05:12.753418 5981 ts_tablet_manager.cc:1367] T 9b88c43cb8704583a3e65c5c35c205f6 P 1c3fdc5272484ac09ef3e5e36a08f27c: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.001s
I20240501 14:05:12.754050 5986 tablet_bootstrap.cc:492] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf: No bootstrap required, opened a new log
I20240501 14:05:12.754212 5986 ts_tablet_manager.cc:1367] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.007s
I20240501 14:05:12.754504 5981 raft_consensus.cc:348] T 9b88c43cb8704583a3e65c5c35c205f6 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:12.754703 5981 raft_consensus.cc:374] T 9b88c43cb8704583a3e65c5c35c205f6 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:12.754837 5981 raft_consensus.cc:729] T 9b88c43cb8704583a3e65c5c35c205f6 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c3fdc5272484ac09ef3e5e36a08f27c, State: Initialized, Role: FOLLOWER
I20240501 14:05:12.755059 5981 consensus_queue.cc:260] T 9b88c43cb8704583a3e65c5c35c205f6 P 1c3fdc5272484ac09ef3e5e36a08f27c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:12.755298 5986 raft_consensus.cc:348] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:12.755527 5986 raft_consensus.cc:374] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:12.755566 5981 ts_tablet_manager.cc:1398] T 9b88c43cb8704583a3e65c5c35c205f6 P 1c3fdc5272484ac09ef3e5e36a08f27c: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:05:12.755622 5986 raft_consensus.cc:729] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be56773105e744919601127a94e98adf, State: Initialized, Role: FOLLOWER
I20240501 14:05:12.755826 5986 consensus_queue.cc:260] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:12.756311 5986 ts_tablet_manager.cc:1398] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:05:12.760341 5989 raft_consensus.cc:482] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:05:12.760533 5989 raft_consensus.cc:504] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } }
I20240501 14:05:12.762256 5989 leader_election.cc:288] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209), 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129:44033)
I20240501 14:05:12.769083 5256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88ed5438fc934adc9fb98c6cb4476598" candidate_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" is_pre_election: true
I20240501 14:05:12.769083 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88ed5438fc934adc9fb98c6cb4476598" candidate_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04088ee3bea440a7bd5ff772152b9410" is_pre_election: true
I20240501 14:05:12.769675 5256 raft_consensus.cc:2454] T 88ed5438fc934adc9fb98c6cb4476598 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9c60449cf3c46f8aca4ab6c535302a4 in term 0.
I20240501 14:05:12.769675 4995 raft_consensus.cc:2454] T 88ed5438fc934adc9fb98c6cb4476598 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9c60449cf3c46f8aca4ab6c535302a4 in term 0.
I20240501 14:05:12.770362 5060 leader_election.cc:302] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 04088ee3bea440a7bd5ff772152b9410, 1c3fdc5272484ac09ef3e5e36a08f27c, e9c60449cf3c46f8aca4ab6c535302a4; no voters:
I20240501 14:05:12.770761 5989 raft_consensus.cc:2781] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:05:12.770895 5989 raft_consensus.cc:482] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:05:12.770982 5989 raft_consensus.cc:3037] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:12.772951 5989 raft_consensus.cc:504] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } }
I20240501 14:05:12.773582 5989 leader_election.cc:288] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 [CANDIDATE]: Term 1 election: Requested vote from peers 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209), 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129:44033)
I20240501 14:05:12.774030 5256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88ed5438fc934adc9fb98c6cb4476598" candidate_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c"
I20240501 14:05:12.774080 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88ed5438fc934adc9fb98c6cb4476598" candidate_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04088ee3bea440a7bd5ff772152b9410"
I20240501 14:05:12.774223 5256 raft_consensus.cc:3037] T 88ed5438fc934adc9fb98c6cb4476598 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:12.774259 4995 raft_consensus.cc:3037] T 88ed5438fc934adc9fb98c6cb4476598 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:12.776227 4995 raft_consensus.cc:2454] T 88ed5438fc934adc9fb98c6cb4476598 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9c60449cf3c46f8aca4ab6c535302a4 in term 1.
I20240501 14:05:12.776423 5256 raft_consensus.cc:2454] T 88ed5438fc934adc9fb98c6cb4476598 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9c60449cf3c46f8aca4ab6c535302a4 in term 1.
I20240501 14:05:12.776605 5060 leader_election.cc:302] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04088ee3bea440a7bd5ff772152b9410, e9c60449cf3c46f8aca4ab6c535302a4; no voters:
I20240501 14:05:12.776911 5989 raft_consensus.cc:2781] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:05:12.777346 5989 raft_consensus.cc:686] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 LEADER]: Becoming Leader. State: Replica: e9c60449cf3c46f8aca4ab6c535302a4, State: Running, Role: LEADER
I20240501 14:05:12.777789 5989 consensus_queue.cc:237] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } }
I20240501 14:05:12.782578 4853 catalog_manager.cc:5484] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130). New cstate: current_term: 1 leader_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } health_report { overall_health: UNKNOWN } } }
W20240501 14:05:12.801903 5432 tablet.cc:2369] T 43b46fcf827c4275a47291afb107c290 P be56773105e744919601127a94e98adf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:05:12.822302 5989 raft_consensus.cc:482] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:05:12.822538 5989 raft_consensus.cc:504] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } }
I20240501 14:05:12.823225 5989 leader_election.cc:288] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be56773105e744919601127a94e98adf (127.29.124.132:35459), 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129:44033)
I20240501 14:05:12.824553 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43b46fcf827c4275a47291afb107c290" candidate_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04088ee3bea440a7bd5ff772152b9410" is_pre_election: true
I20240501 14:05:12.824785 4995 raft_consensus.cc:2454] T 43b46fcf827c4275a47291afb107c290 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9c60449cf3c46f8aca4ab6c535302a4 in term 0.
I20240501 14:05:12.825178 5060 leader_election.cc:302] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04088ee3bea440a7bd5ff772152b9410, e9c60449cf3c46f8aca4ab6c535302a4; no voters:
I20240501 14:05:12.825469 5989 raft_consensus.cc:2781] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:05:12.825570 5989 raft_consensus.cc:482] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:05:12.825668 5989 raft_consensus.cc:3037] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:12.827936 5989 raft_consensus.cc:504] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } }
I20240501 14:05:12.828580 5989 leader_election.cc:288] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 [CANDIDATE]: Term 1 election: Requested vote from peers be56773105e744919601127a94e98adf (127.29.124.132:35459), 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129:44033)
I20240501 14:05:12.828976 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43b46fcf827c4275a47291afb107c290" candidate_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04088ee3bea440a7bd5ff772152b9410"
I20240501 14:05:12.829197 4995 raft_consensus.cc:3037] T 43b46fcf827c4275a47291afb107c290 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:12.829541 5385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43b46fcf827c4275a47291afb107c290" candidate_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf"
I20240501 14:05:12.829592 5386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "43b46fcf827c4275a47291afb107c290" candidate_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf" is_pre_election: true
I20240501 14:05:12.829982 5385 raft_consensus.cc:3037] T 43b46fcf827c4275a47291afb107c290 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:12.831616 4995 raft_consensus.cc:2454] T 43b46fcf827c4275a47291afb107c290 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9c60449cf3c46f8aca4ab6c535302a4 in term 1.
I20240501 14:05:12.831991 5060 leader_election.cc:302] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04088ee3bea440a7bd5ff772152b9410, e9c60449cf3c46f8aca4ab6c535302a4; no voters:
I20240501 14:05:12.832209 5385 raft_consensus.cc:2454] T 43b46fcf827c4275a47291afb107c290 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9c60449cf3c46f8aca4ab6c535302a4 in term 1.
I20240501 14:05:12.832271 5989 raft_consensus.cc:2781] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:05:12.832434 5989 raft_consensus.cc:686] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 LEADER]: Becoming Leader. State: Replica: e9c60449cf3c46f8aca4ab6c535302a4, State: Running, Role: LEADER
I20240501 14:05:12.832681 5989 consensus_queue.cc:237] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } }
I20240501 14:05:12.835283 4853 catalog_manager.cc:5484] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130). New cstate: current_term: 1 leader_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } health_report { overall_health: UNKNOWN } } }
W20240501 14:05:12.856829 5302 tablet.cc:2369] T 88ed5438fc934adc9fb98c6cb4476598 P 1c3fdc5272484ac09ef3e5e36a08f27c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:05:12.868119 5993 raft_consensus.cc:482] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:05:12.868335 5993 raft_consensus.cc:504] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:12.870088 5993 leader_election.cc:288] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129:44033), 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209)
I20240501 14:05:12.876840 5256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b88c43cb8704583a3e65c5c35c205f6" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" is_pre_election: true
I20240501 14:05:12.876856 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b88c43cb8704583a3e65c5c35c205f6" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04088ee3bea440a7bd5ff772152b9410" is_pre_election: true
I20240501 14:05:12.877067 4995 raft_consensus.cc:2454] T 9b88c43cb8704583a3e65c5c35c205f6 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 0.
I20240501 14:05:12.877075 5256 raft_consensus.cc:2454] T 9b88c43cb8704583a3e65c5c35c205f6 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 0.
I20240501 14:05:12.877563 5320 leader_election.cc:302] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04088ee3bea440a7bd5ff772152b9410, be56773105e744919601127a94e98adf; no voters:
I20240501 14:05:12.877940 5993 raft_consensus.cc:2781] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:05:12.878086 5993 raft_consensus.cc:482] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:05:12.878167 5993 raft_consensus.cc:3037] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:12.880129 5993 raft_consensus.cc:504] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:12.880715 5993 leader_election.cc:288] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 1 election: Requested vote from peers 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129:44033), 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209)
I20240501 14:05:12.881101 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b88c43cb8704583a3e65c5c35c205f6" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04088ee3bea440a7bd5ff772152b9410"
I20240501 14:05:12.881170 5256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b88c43cb8704583a3e65c5c35c205f6" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c"
I20240501 14:05:12.881301 4995 raft_consensus.cc:3037] T 9b88c43cb8704583a3e65c5c35c205f6 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:12.881412 5256 raft_consensus.cc:3037] T 9b88c43cb8704583a3e65c5c35c205f6 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:12.883139 4995 raft_consensus.cc:2454] T 9b88c43cb8704583a3e65c5c35c205f6 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 1.
I20240501 14:05:12.883205 5256 raft_consensus.cc:2454] T 9b88c43cb8704583a3e65c5c35c205f6 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 1.
I20240501 14:05:12.883534 5320 leader_election.cc:302] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04088ee3bea440a7bd5ff772152b9410, be56773105e744919601127a94e98adf; no voters:
I20240501 14:05:12.883831 5993 raft_consensus.cc:2781] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:05:12.884248 5993 raft_consensus.cc:686] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf [term 1 LEADER]: Becoming Leader. State: Replica: be56773105e744919601127a94e98adf, State: Running, Role: LEADER
I20240501 14:05:12.884680 5993 consensus_queue.cc:237] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:12.889036 4853 catalog_manager.cc:5484] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf reported cstate change: term changed from 0 to 1, leader changed from <none> to be56773105e744919601127a94e98adf (127.29.124.132). New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } health_report { overall_health: UNKNOWN } } }
W20240501 14:05:12.908701 5172 tablet.cc:2369] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240501 14:05:12.965744 5041 tablet.cc:2369] T 43b46fcf827c4275a47291afb107c290 P 04088ee3bea440a7bd5ff772152b9410: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240501 14:05:13.234454 5989 consensus_queue.cc:1035] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:05:13.236864 6006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:05:13.237188 6006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 14:05:13.243050 6012 consensus_queue.cc:1035] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:05:13.249168 6006 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:05:13.268663 5993 consensus_queue.cc:1035] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:05:13.274484 6004 consensus_queue.cc:1035] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'
I20240501 14:05:13.299450 6012 consensus_queue.cc:1035] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:05:13.304508 5989 consensus_queue.cc:1035] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 258.505 ms
time per row: 0.129252 ms
I20240501 14:05:13.660580 4846 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55620:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:05:13.669836 5236 tablet_service.cc:1467] Processing CreateTablet for tablet 1a7bbfc15a76450bbc9197037e856e08 (DEFAULT_TABLE table=rebalance_test_table_1 [id=403fdb578f8e4066a56f685925e83936]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:13.670061 5106 tablet_service.cc:1467] Processing CreateTablet for tablet 1a7bbfc15a76450bbc9197037e856e08 (DEFAULT_TABLE table=rebalance_test_table_1 [id=403fdb578f8e4066a56f685925e83936]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:13.670372 5236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a7bbfc15a76450bbc9197037e856e08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:13.670493 5106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a7bbfc15a76450bbc9197037e856e08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:13.670873 5235 tablet_service.cc:1467] Processing CreateTablet for tablet f4d2b8c005b64bbb8e6621da068d15e5 (DEFAULT_TABLE table=rebalance_test_table_1 [id=403fdb578f8e4066a56f685925e83936]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:13.671269 5235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4d2b8c005b64bbb8e6621da068d15e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:13.675469 5365 tablet_service.cc:1467] Processing CreateTablet for tablet 1a7bbfc15a76450bbc9197037e856e08 (DEFAULT_TABLE table=rebalance_test_table_1 [id=403fdb578f8e4066a56f685925e83936]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:13.675516 5234 tablet_service.cc:1467] Processing CreateTablet for tablet c8a961e711114f369ba06840c062b982 (DEFAULT_TABLE table=rebalance_test_table_1 [id=403fdb578f8e4066a56f685925e83936]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:13.675549 4974 tablet_service.cc:1467] Processing CreateTablet for tablet c8a961e711114f369ba06840c062b982 (DEFAULT_TABLE table=rebalance_test_table_1 [id=403fdb578f8e4066a56f685925e83936]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:13.675920 5365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a7bbfc15a76450bbc9197037e856e08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:13.675935 5234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8a961e711114f369ba06840c062b982. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:13.675957 4974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8a961e711114f369ba06840c062b982. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:13.677649 5366 tablet_service.cc:1467] Processing CreateTablet for tablet f4d2b8c005b64bbb8e6621da068d15e5 (DEFAULT_TABLE table=rebalance_test_table_1 [id=403fdb578f8e4066a56f685925e83936]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:13.678038 5366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4d2b8c005b64bbb8e6621da068d15e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:13.682724 5105 tablet_service.cc:1467] Processing CreateTablet for tablet f4d2b8c005b64bbb8e6621da068d15e5 (DEFAULT_TABLE table=rebalance_test_table_1 [id=403fdb578f8e4066a56f685925e83936]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:13.683243 5105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4d2b8c005b64bbb8e6621da068d15e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:13.682775 5104 tablet_service.cc:1467] Processing CreateTablet for tablet c8a961e711114f369ba06840c062b982 (DEFAULT_TABLE table=rebalance_test_table_1 [id=403fdb578f8e4066a56f685925e83936]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:13.685969 5104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8a961e711114f369ba06840c062b982. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:13.695922 6073 tablet_bootstrap.cc:492] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf: Bootstrap starting.
I20240501 14:05:13.697808 6072 tablet_bootstrap.cc:492] T c8a961e711114f369ba06840c062b982 P 04088ee3bea440a7bd5ff772152b9410: Bootstrap starting.
I20240501 14:05:13.698647 6073 tablet_bootstrap.cc:654] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:13.699949 6071 tablet_bootstrap.cc:492] T 1a7bbfc15a76450bbc9197037e856e08 P e9c60449cf3c46f8aca4ab6c535302a4: Bootstrap starting.
I20240501 14:05:13.700029 6072 tablet_bootstrap.cc:654] T c8a961e711114f369ba06840c062b982 P 04088ee3bea440a7bd5ff772152b9410: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:13.701872 6071 tablet_bootstrap.cc:654] T 1a7bbfc15a76450bbc9197037e856e08 P e9c60449cf3c46f8aca4ab6c535302a4: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:13.702271 6072 tablet_bootstrap.cc:492] T c8a961e711114f369ba06840c062b982 P 04088ee3bea440a7bd5ff772152b9410: No bootstrap required, opened a new log
I20240501 14:05:13.702426 6072 ts_tablet_manager.cc:1367] T c8a961e711114f369ba06840c062b982 P 04088ee3bea440a7bd5ff772152b9410: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:05:13.704392 6072 raft_consensus.cc:348] T c8a961e711114f369ba06840c062b982 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:13.704627 6072 raft_consensus.cc:374] T c8a961e711114f369ba06840c062b982 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:13.704715 6072 raft_consensus.cc:729] T c8a961e711114f369ba06840c062b982 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04088ee3bea440a7bd5ff772152b9410, State: Initialized, Role: FOLLOWER
I20240501 14:05:13.704958 6072 consensus_queue.cc:260] T c8a961e711114f369ba06840c062b982 P 04088ee3bea440a7bd5ff772152b9410 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:13.705556 6072 ts_tablet_manager.cc:1398] T c8a961e711114f369ba06840c062b982 P 04088ee3bea440a7bd5ff772152b9410: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 14:05:13.706141 6070 tablet_bootstrap.cc:492] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c: Bootstrap starting.
I20240501 14:05:13.707721 6070 tablet_bootstrap.cc:654] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:13.709553 6073 tablet_bootstrap.cc:492] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf: No bootstrap required, opened a new log
I20240501 14:05:13.709777 6073 ts_tablet_manager.cc:1367] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.005s
I20240501 14:05:13.710870 6073 raft_consensus.cc:348] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:13.711109 6073 raft_consensus.cc:374] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:13.711246 6073 raft_consensus.cc:729] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be56773105e744919601127a94e98adf, State: Initialized, Role: FOLLOWER
I20240501 14:05:13.711547 6073 consensus_queue.cc:260] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:13.711824 6071 tablet_bootstrap.cc:492] T 1a7bbfc15a76450bbc9197037e856e08 P e9c60449cf3c46f8aca4ab6c535302a4: No bootstrap required, opened a new log
I20240501 14:05:13.711969 6071 ts_tablet_manager.cc:1367] T 1a7bbfc15a76450bbc9197037e856e08 P e9c60449cf3c46f8aca4ab6c535302a4: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.005s
I20240501 14:05:13.712652 6071 raft_consensus.cc:348] T 1a7bbfc15a76450bbc9197037e856e08 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:13.712785 6073 ts_tablet_manager.cc:1398] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:05:13.712921 6071 raft_consensus.cc:374] T 1a7bbfc15a76450bbc9197037e856e08 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:13.713073 6073 tablet_bootstrap.cc:492] T 1a7bbfc15a76450bbc9197037e856e08 P be56773105e744919601127a94e98adf: Bootstrap starting.
I20240501 14:05:13.713074 6071 raft_consensus.cc:729] T 1a7bbfc15a76450bbc9197037e856e08 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9c60449cf3c46f8aca4ab6c535302a4, State: Initialized, Role: FOLLOWER
I20240501 14:05:13.713419 6071 consensus_queue.cc:260] T 1a7bbfc15a76450bbc9197037e856e08 P e9c60449cf3c46f8aca4ab6c535302a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:13.714913 6073 tablet_bootstrap.cc:654] T 1a7bbfc15a76450bbc9197037e856e08 P be56773105e744919601127a94e98adf: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:13.714928 6071 ts_tablet_manager.cc:1398] T 1a7bbfc15a76450bbc9197037e856e08 P e9c60449cf3c46f8aca4ab6c535302a4: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:05:13.715214 6071 tablet_bootstrap.cc:492] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4: Bootstrap starting.
I20240501 14:05:13.717002 6071 tablet_bootstrap.cc:654] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:13.717908 6070 tablet_bootstrap.cc:492] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c: No bootstrap required, opened a new log
I20240501 14:05:13.718129 6070 ts_tablet_manager.cc:1367] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.008s
I20240501 14:05:13.719065 6070 raft_consensus.cc:348] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:13.719362 6070 raft_consensus.cc:374] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:13.719463 6070 raft_consensus.cc:729] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c3fdc5272484ac09ef3e5e36a08f27c, State: Initialized, Role: FOLLOWER
I20240501 14:05:13.719731 6070 consensus_queue.cc:260] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:13.720281 6070 ts_tablet_manager.cc:1398] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:05:13.720606 6070 tablet_bootstrap.cc:492] T c8a961e711114f369ba06840c062b982 P 1c3fdc5272484ac09ef3e5e36a08f27c: Bootstrap starting.
I20240501 14:05:13.721378 6073 tablet_bootstrap.cc:492] T 1a7bbfc15a76450bbc9197037e856e08 P be56773105e744919601127a94e98adf: No bootstrap required, opened a new log
I20240501 14:05:13.721599 6073 ts_tablet_manager.cc:1367] T 1a7bbfc15a76450bbc9197037e856e08 P be56773105e744919601127a94e98adf: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20240501 14:05:13.722218 6070 tablet_bootstrap.cc:654] T c8a961e711114f369ba06840c062b982 P 1c3fdc5272484ac09ef3e5e36a08f27c: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:13.722591 6073 raft_consensus.cc:348] T 1a7bbfc15a76450bbc9197037e856e08 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:13.723093 6073 raft_consensus.cc:374] T 1a7bbfc15a76450bbc9197037e856e08 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:13.723225 6073 raft_consensus.cc:729] T 1a7bbfc15a76450bbc9197037e856e08 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be56773105e744919601127a94e98adf, State: Initialized, Role: FOLLOWER
I20240501 14:05:13.723593 6073 consensus_queue.cc:260] T 1a7bbfc15a76450bbc9197037e856e08 P be56773105e744919601127a94e98adf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:13.725384 6073 ts_tablet_manager.cc:1398] T 1a7bbfc15a76450bbc9197037e856e08 P be56773105e744919601127a94e98adf: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:05:13.726614 6070 tablet_bootstrap.cc:492] T c8a961e711114f369ba06840c062b982 P 1c3fdc5272484ac09ef3e5e36a08f27c: No bootstrap required, opened a new log
I20240501 14:05:13.726615 6071 tablet_bootstrap.cc:492] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4: No bootstrap required, opened a new log
I20240501 14:05:13.726778 6071 ts_tablet_manager.cc:1367] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.001s
I20240501 14:05:13.726778 6070 ts_tablet_manager.cc:1367] T c8a961e711114f369ba06840c062b982 P 1c3fdc5272484ac09ef3e5e36a08f27c: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20240501 14:05:13.727499 6070 raft_consensus.cc:348] T c8a961e711114f369ba06840c062b982 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:13.727532 6071 raft_consensus.cc:348] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:13.727731 6071 raft_consensus.cc:374] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:13.727731 6070 raft_consensus.cc:374] T c8a961e711114f369ba06840c062b982 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:13.727806 6070 raft_consensus.cc:729] T c8a961e711114f369ba06840c062b982 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c3fdc5272484ac09ef3e5e36a08f27c, State: Initialized, Role: FOLLOWER
I20240501 14:05:13.727847 6071 raft_consensus.cc:729] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9c60449cf3c46f8aca4ab6c535302a4, State: Initialized, Role: FOLLOWER
I20240501 14:05:13.728097 6070 consensus_queue.cc:260] T c8a961e711114f369ba06840c062b982 P 1c3fdc5272484ac09ef3e5e36a08f27c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:13.728099 6071 consensus_queue.cc:260] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:13.728608 6070 ts_tablet_manager.cc:1398] T c8a961e711114f369ba06840c062b982 P 1c3fdc5272484ac09ef3e5e36a08f27c: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:05:13.728893 6070 tablet_bootstrap.cc:492] T f4d2b8c005b64bbb8e6621da068d15e5 P 1c3fdc5272484ac09ef3e5e36a08f27c: Bootstrap starting.
I20240501 14:05:13.729434 6071 ts_tablet_manager.cc:1398] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:05:13.729728 6071 tablet_bootstrap.cc:492] T f4d2b8c005b64bbb8e6621da068d15e5 P e9c60449cf3c46f8aca4ab6c535302a4: Bootstrap starting.
I20240501 14:05:13.731307 6071 tablet_bootstrap.cc:654] T f4d2b8c005b64bbb8e6621da068d15e5 P e9c60449cf3c46f8aca4ab6c535302a4: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:13.731420 6070 tablet_bootstrap.cc:654] T f4d2b8c005b64bbb8e6621da068d15e5 P 1c3fdc5272484ac09ef3e5e36a08f27c: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:13.733721 6070 tablet_bootstrap.cc:492] T f4d2b8c005b64bbb8e6621da068d15e5 P 1c3fdc5272484ac09ef3e5e36a08f27c: No bootstrap required, opened a new log
I20240501 14:05:13.733860 6070 ts_tablet_manager.cc:1367] T f4d2b8c005b64bbb8e6621da068d15e5 P 1c3fdc5272484ac09ef3e5e36a08f27c: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.002s
I20240501 14:05:13.734129 6071 tablet_bootstrap.cc:492] T f4d2b8c005b64bbb8e6621da068d15e5 P e9c60449cf3c46f8aca4ab6c535302a4: No bootstrap required, opened a new log
I20240501 14:05:13.734262 6071 ts_tablet_manager.cc:1367] T f4d2b8c005b64bbb8e6621da068d15e5 P e9c60449cf3c46f8aca4ab6c535302a4: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240501 14:05:13.734611 6070 raft_consensus.cc:348] T f4d2b8c005b64bbb8e6621da068d15e5 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:13.734813 6070 raft_consensus.cc:374] T f4d2b8c005b64bbb8e6621da068d15e5 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:13.734910 6070 raft_consensus.cc:729] T f4d2b8c005b64bbb8e6621da068d15e5 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c3fdc5272484ac09ef3e5e36a08f27c, State: Initialized, Role: FOLLOWER
I20240501 14:05:13.734946 6071 raft_consensus.cc:348] T f4d2b8c005b64bbb8e6621da068d15e5 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:13.735124 6071 raft_consensus.cc:374] T f4d2b8c005b64bbb8e6621da068d15e5 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:13.735217 6071 raft_consensus.cc:729] T f4d2b8c005b64bbb8e6621da068d15e5 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9c60449cf3c46f8aca4ab6c535302a4, State: Initialized, Role: FOLLOWER
I20240501 14:05:13.735150 6070 consensus_queue.cc:260] T f4d2b8c005b64bbb8e6621da068d15e5 P 1c3fdc5272484ac09ef3e5e36a08f27c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:13.735441 6071 consensus_queue.cc:260] T f4d2b8c005b64bbb8e6621da068d15e5 P e9c60449cf3c46f8aca4ab6c535302a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:13.735826 6070 ts_tablet_manager.cc:1398] T f4d2b8c005b64bbb8e6621da068d15e5 P 1c3fdc5272484ac09ef3e5e36a08f27c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:05:13.735994 6071 ts_tablet_manager.cc:1398] T f4d2b8c005b64bbb8e6621da068d15e5 P e9c60449cf3c46f8aca4ab6c535302a4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:05:13.742789 5993 raft_consensus.cc:482] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:05:13.742957 5993 raft_consensus.cc:504] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:13.743680 5993 leader_election.cc:288] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377), 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209)
I20240501 14:05:13.744199 5255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4d2b8c005b64bbb8e6621da068d15e5" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" is_pre_election: true
I20240501 14:05:13.744408 5255 raft_consensus.cc:2454] T f4d2b8c005b64bbb8e6621da068d15e5 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 0.
I20240501 14:05:13.744773 5319 leader_election.cc:302] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters:
I20240501 14:05:13.745054 5993 raft_consensus.cc:2781] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:05:13.745172 5993 raft_consensus.cc:482] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:05:13.745345 5993 raft_consensus.cc:3037] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:13.747232 5993 raft_consensus.cc:504] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:13.748023 5993 leader_election.cc:288] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 1 election: Requested vote from peers e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377), 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209)
I20240501 14:05:13.748289 5255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4d2b8c005b64bbb8e6621da068d15e5" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c"
I20240501 14:05:13.748483 5255 raft_consensus.cc:3037] T f4d2b8c005b64bbb8e6621da068d15e5 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:13.750694 5126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4d2b8c005b64bbb8e6621da068d15e5" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" is_pre_election: true
I20240501 14:05:13.750720 5125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f4d2b8c005b64bbb8e6621da068d15e5" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9c60449cf3c46f8aca4ab6c535302a4"
I20240501 14:05:13.750883 5255 raft_consensus.cc:2454] T f4d2b8c005b64bbb8e6621da068d15e5 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 1.
I20240501 14:05:13.751048 5126 raft_consensus.cc:2454] T f4d2b8c005b64bbb8e6621da068d15e5 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 0.
I20240501 14:05:13.751366 5319 leader_election.cc:302] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters:
I20240501 14:05:13.752209 5993 raft_consensus.cc:2781] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:05:13.752373 5993 raft_consensus.cc:686] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf [term 1 LEADER]: Becoming Leader. State: Replica: be56773105e744919601127a94e98adf, State: Running, Role: LEADER
I20240501 14:05:13.752674 5993 consensus_queue.cc:237] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:13.755316 4846 catalog_manager.cc:5484] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf reported cstate change: term changed from 0 to 1, leader changed from <none> to be56773105e744919601127a94e98adf (127.29.124.132). New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } health_report { overall_health: UNKNOWN } } }
I20240501 14:05:13.762429 5989 raft_consensus.cc:482] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:05:13.762598 5989 raft_consensus.cc:504] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:13.763231 5989 leader_election.cc:288] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129:44033), 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209)
I20240501 14:05:13.763751 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8a961e711114f369ba06840c062b982" candidate_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04088ee3bea440a7bd5ff772152b9410" is_pre_election: true
I20240501 14:05:13.763756 5255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8a961e711114f369ba06840c062b982" candidate_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" is_pre_election: true
I20240501 14:05:13.763973 5255 raft_consensus.cc:2454] T c8a961e711114f369ba06840c062b982 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9c60449cf3c46f8aca4ab6c535302a4 in term 0.
I20240501 14:05:13.763989 4995 raft_consensus.cc:2454] T c8a961e711114f369ba06840c062b982 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9c60449cf3c46f8aca4ab6c535302a4 in term 0.
I20240501 14:05:13.764395 5059 leader_election.cc:302] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, e9c60449cf3c46f8aca4ab6c535302a4; no voters:
I20240501 14:05:13.764647 5989 raft_consensus.cc:2781] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:05:13.764755 5989 raft_consensus.cc:482] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:05:13.764835 5989 raft_consensus.cc:3037] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:13.766716 5989 raft_consensus.cc:504] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:13.767264 5989 leader_election.cc:288] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 [CANDIDATE]: Term 1 election: Requested vote from peers 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129:44033), 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209)
I20240501 14:05:13.767776 5255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8a961e711114f369ba06840c062b982" candidate_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c"
I20240501 14:05:13.767776 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c8a961e711114f369ba06840c062b982" candidate_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04088ee3bea440a7bd5ff772152b9410"
I20240501 14:05:13.768004 5255 raft_consensus.cc:3037] T c8a961e711114f369ba06840c062b982 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:13.768002 4995 raft_consensus.cc:3037] T c8a961e711114f369ba06840c062b982 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:13.770745 4995 raft_consensus.cc:2454] T c8a961e711114f369ba06840c062b982 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9c60449cf3c46f8aca4ab6c535302a4 in term 1.
I20240501 14:05:13.770745 5255 raft_consensus.cc:2454] T c8a961e711114f369ba06840c062b982 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9c60449cf3c46f8aca4ab6c535302a4 in term 1.
I20240501 14:05:13.771238 5059 leader_election.cc:302] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 04088ee3bea440a7bd5ff772152b9410, 1c3fdc5272484ac09ef3e5e36a08f27c, e9c60449cf3c46f8aca4ab6c535302a4; no voters:
I20240501 14:05:13.771548 5989 raft_consensus.cc:2781] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:05:13.771692 5989 raft_consensus.cc:686] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 LEADER]: Becoming Leader. State: Replica: e9c60449cf3c46f8aca4ab6c535302a4, State: Running, Role: LEADER
I20240501 14:05:13.771989 5989 consensus_queue.cc:237] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:13.774636 4846 catalog_manager.cc:5484] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130). New cstate: current_term: 1 leader_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } health_report { overall_health: UNKNOWN } } }
I20240501 14:05:14.030223 5992 raft_consensus.cc:482] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:05:14.030433 5992 raft_consensus.cc:504] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:14.031951 5992 leader_election.cc:288] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377), be56773105e744919601127a94e98adf (127.29.124.132:35459)
I20240501 14:05:14.038520 5125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a7bbfc15a76450bbc9197037e856e08" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" is_pre_election: true
I20240501 14:05:14.038679 5385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a7bbfc15a76450bbc9197037e856e08" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf" is_pre_election: true
I20240501 14:05:14.038767 5125 raft_consensus.cc:2454] T 1a7bbfc15a76450bbc9197037e856e08 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 0.
I20240501 14:05:14.038908 5385 raft_consensus.cc:2454] T 1a7bbfc15a76450bbc9197037e856e08 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 0.
I20240501 14:05:14.039196 5190 leader_election.cc:302] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, e9c60449cf3c46f8aca4ab6c535302a4; no voters:
I20240501 14:05:14.039615 5992 raft_consensus.cc:2781] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:05:14.039727 5992 raft_consensus.cc:482] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:05:14.039831 5992 raft_consensus.cc:3037] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:14.041934 5992 raft_consensus.cc:504] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:14.042646 5992 leader_election.cc:288] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 1 election: Requested vote from peers e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377), be56773105e744919601127a94e98adf (127.29.124.132:35459)
I20240501 14:05:14.043193 5125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a7bbfc15a76450bbc9197037e856e08" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9c60449cf3c46f8aca4ab6c535302a4"
I20240501 14:05:14.043424 5125 raft_consensus.cc:3037] T 1a7bbfc15a76450bbc9197037e856e08 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:14.043913 5385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a7bbfc15a76450bbc9197037e856e08" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf"
I20240501 14:05:14.044123 5385 raft_consensus.cc:3037] T 1a7bbfc15a76450bbc9197037e856e08 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:14.045748 5125 raft_consensus.cc:2454] T 1a7bbfc15a76450bbc9197037e856e08 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 1.
I20240501 14:05:14.046172 5190 leader_election.cc:302] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, e9c60449cf3c46f8aca4ab6c535302a4; no voters:
I20240501 14:05:14.046301 5385 raft_consensus.cc:2454] T 1a7bbfc15a76450bbc9197037e856e08 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 1.
I20240501 14:05:14.046473 5992 raft_consensus.cc:2781] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:05:14.046877 5992 raft_consensus.cc:686] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEADER]: Becoming Leader. State: Replica: 1c3fdc5272484ac09ef3e5e36a08f27c, State: Running, Role: LEADER
I20240501 14:05:14.047435 5992 consensus_queue.cc:237] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:14.050916 4846 catalog_manager.cc:5484] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131). New cstate: current_term: 1 leader_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } health_report { overall_health: UNKNOWN } } }
I20240501 14:05:14.224210 5989 consensus_queue.cc:1035] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:05:14.232908 5993 consensus_queue.cc:1035] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:05:14.235656 5999 consensus_queue.cc:1035] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:05:14.247483 5125 raft_consensus.cc:3037] T f4d2b8c005b64bbb8e6621da068d15e5 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:14.250385 5993 consensus_queue.cc:1035] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:05:14.486346 6091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:05:14.486704 6091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:05:14.494148 6091 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
I20240501 14:05:14.527025 5992 consensus_queue.cc:1035] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:05:14.532968 6089 consensus_queue.cc:1035] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 237.497 ms
time per row: 0.118749 ms
I20240501 14:05:14.876699 4847 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55620:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:05:14.885354 5106 tablet_service.cc:1467] Processing CreateTablet for tablet a726769d24284fedadd8e9e63cd27d05 (DEFAULT_TABLE table=rebalance_test_table_2 [id=facc024bfd644356a8879fd9661c928d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:14.885411 4974 tablet_service.cc:1467] Processing CreateTablet for tablet a726769d24284fedadd8e9e63cd27d05 (DEFAULT_TABLE table=rebalance_test_table_2 [id=facc024bfd644356a8879fd9661c928d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:14.885851 4974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a726769d24284fedadd8e9e63cd27d05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:14.885855 5106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a726769d24284fedadd8e9e63cd27d05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:14.885906 5366 tablet_service.cc:1467] Processing CreateTablet for tablet a726769d24284fedadd8e9e63cd27d05 (DEFAULT_TABLE table=rebalance_test_table_2 [id=facc024bfd644356a8879fd9661c928d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:14.886291 5366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a726769d24284fedadd8e9e63cd27d05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:14.886780 5236 tablet_service.cc:1467] Processing CreateTablet for tablet 71660575d94748d8a9329f909a42a0cd (DEFAULT_TABLE table=rebalance_test_table_2 [id=facc024bfd644356a8879fd9661c928d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:14.887162 5236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71660575d94748d8a9329f909a42a0cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:14.893186 5365 tablet_service.cc:1467] Processing CreateTablet for tablet 71660575d94748d8a9329f909a42a0cd (DEFAULT_TABLE table=rebalance_test_table_2 [id=facc024bfd644356a8879fd9661c928d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:14.893604 5365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71660575d94748d8a9329f909a42a0cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:14.893620 4975 tablet_service.cc:1467] Processing CreateTablet for tablet 71660575d94748d8a9329f909a42a0cd (DEFAULT_TABLE table=rebalance_test_table_2 [id=facc024bfd644356a8879fd9661c928d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:14.893996 4975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71660575d94748d8a9329f909a42a0cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:14.897748 5364 tablet_service.cc:1467] Processing CreateTablet for tablet fee16905387d428aa02cd2070c86c790 (DEFAULT_TABLE table=rebalance_test_table_2 [id=facc024bfd644356a8879fd9661c928d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:14.898171 5364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fee16905387d428aa02cd2070c86c790. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:14.898434 5105 tablet_service.cc:1467] Processing CreateTablet for tablet fee16905387d428aa02cd2070c86c790 (DEFAULT_TABLE table=rebalance_test_table_2 [id=facc024bfd644356a8879fd9661c928d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:14.898840 5105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fee16905387d428aa02cd2070c86c790. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:14.900965 6158 tablet_bootstrap.cc:492] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4: Bootstrap starting.
I20240501 14:05:14.901944 4973 tablet_service.cc:1467] Processing CreateTablet for tablet fee16905387d428aa02cd2070c86c790 (DEFAULT_TABLE table=rebalance_test_table_2 [id=facc024bfd644356a8879fd9661c928d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:14.902352 4973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fee16905387d428aa02cd2070c86c790. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:14.903061 6158 tablet_bootstrap.cc:654] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:14.907723 6159 tablet_bootstrap.cc:492] T a726769d24284fedadd8e9e63cd27d05 P be56773105e744919601127a94e98adf: Bootstrap starting.
I20240501 14:05:14.909413 6159 tablet_bootstrap.cc:654] T a726769d24284fedadd8e9e63cd27d05 P be56773105e744919601127a94e98adf: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:14.911322 6158 tablet_bootstrap.cc:492] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4: No bootstrap required, opened a new log
I20240501 14:05:14.911527 6160 tablet_bootstrap.cc:492] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c: Bootstrap starting.
I20240501 14:05:14.911540 6158 ts_tablet_manager.cc:1367] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.010s
I20240501 14:05:14.912320 6158 raft_consensus.cc:348] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:14.912588 6158 raft_consensus.cc:374] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:14.912703 6158 raft_consensus.cc:729] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9c60449cf3c46f8aca4ab6c535302a4, State: Initialized, Role: FOLLOWER
I20240501 14:05:14.912984 6158 consensus_queue.cc:260] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:14.913642 6158 ts_tablet_manager.cc:1398] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:05:14.913677 6160 tablet_bootstrap.cc:654] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:14.920929 6158 tablet_bootstrap.cc:492] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4: Bootstrap starting.
I20240501 14:05:14.922509 6157 tablet_bootstrap.cc:492] T a726769d24284fedadd8e9e63cd27d05 P 04088ee3bea440a7bd5ff772152b9410: Bootstrap starting.
I20240501 14:05:14.922724 6159 tablet_bootstrap.cc:492] T a726769d24284fedadd8e9e63cd27d05 P be56773105e744919601127a94e98adf: No bootstrap required, opened a new log
I20240501 14:05:14.922907 6158 tablet_bootstrap.cc:654] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:14.923404 6159 ts_tablet_manager.cc:1367] T a726769d24284fedadd8e9e63cd27d05 P be56773105e744919601127a94e98adf: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.007s
I20240501 14:05:14.924394 6160 tablet_bootstrap.cc:492] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c: No bootstrap required, opened a new log
I20240501 14:05:14.924325 6159 raft_consensus.cc:348] T a726769d24284fedadd8e9e63cd27d05 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:14.924583 6160 ts_tablet_manager.cc:1367] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.011s
I20240501 14:05:14.924630 6159 raft_consensus.cc:374] T a726769d24284fedadd8e9e63cd27d05 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:14.924769 6159 raft_consensus.cc:729] T a726769d24284fedadd8e9e63cd27d05 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be56773105e744919601127a94e98adf, State: Initialized, Role: FOLLOWER
I20240501 14:05:14.924875 6157 tablet_bootstrap.cc:654] T a726769d24284fedadd8e9e63cd27d05 P 04088ee3bea440a7bd5ff772152b9410: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:14.925113 6159 consensus_queue.cc:260] T a726769d24284fedadd8e9e63cd27d05 P be56773105e744919601127a94e98adf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:14.925422 6160 raft_consensus.cc:348] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:14.925666 6160 raft_consensus.cc:374] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:14.925796 6160 raft_consensus.cc:729] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c3fdc5272484ac09ef3e5e36a08f27c, State: Initialized, Role: FOLLOWER
I20240501 14:05:14.926131 6160 consensus_queue.cc:260] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:14.926873 6160 ts_tablet_manager.cc:1398] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:05:14.928088 6157 tablet_bootstrap.cc:492] T a726769d24284fedadd8e9e63cd27d05 P 04088ee3bea440a7bd5ff772152b9410: No bootstrap required, opened a new log
I20240501 14:05:14.928300 6157 ts_tablet_manager.cc:1367] T a726769d24284fedadd8e9e63cd27d05 P 04088ee3bea440a7bd5ff772152b9410: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20240501 14:05:14.928925 6159 ts_tablet_manager.cc:1398] T a726769d24284fedadd8e9e63cd27d05 P be56773105e744919601127a94e98adf: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:05:14.929078 6157 raft_consensus.cc:348] T a726769d24284fedadd8e9e63cd27d05 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:14.929329 6157 raft_consensus.cc:374] T a726769d24284fedadd8e9e63cd27d05 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:14.929445 6157 raft_consensus.cc:729] T a726769d24284fedadd8e9e63cd27d05 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04088ee3bea440a7bd5ff772152b9410, State: Initialized, Role: FOLLOWER
I20240501 14:05:14.929672 6157 consensus_queue.cc:260] T a726769d24284fedadd8e9e63cd27d05 P 04088ee3bea440a7bd5ff772152b9410 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:14.930292 6159 tablet_bootstrap.cc:492] T fee16905387d428aa02cd2070c86c790 P be56773105e744919601127a94e98adf: Bootstrap starting.
I20240501 14:05:14.932492 6159 tablet_bootstrap.cc:654] T fee16905387d428aa02cd2070c86c790 P be56773105e744919601127a94e98adf: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:14.933059 6157 ts_tablet_manager.cc:1398] T a726769d24284fedadd8e9e63cd27d05 P 04088ee3bea440a7bd5ff772152b9410: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20240501 14:05:14.933413 6157 tablet_bootstrap.cc:492] T 71660575d94748d8a9329f909a42a0cd P 04088ee3bea440a7bd5ff772152b9410: Bootstrap starting.
I20240501 14:05:14.934073 6158 tablet_bootstrap.cc:492] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4: No bootstrap required, opened a new log
I20240501 14:05:14.934298 6158 ts_tablet_manager.cc:1367] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.006s
I20240501 14:05:14.934943 6157 tablet_bootstrap.cc:654] T 71660575d94748d8a9329f909a42a0cd P 04088ee3bea440a7bd5ff772152b9410: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:14.935153 6158 raft_consensus.cc:348] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } }
I20240501 14:05:14.935436 6158 raft_consensus.cc:374] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:14.935555 6158 raft_consensus.cc:729] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9c60449cf3c46f8aca4ab6c535302a4, State: Initialized, Role: FOLLOWER
I20240501 14:05:14.935874 6158 consensus_queue.cc:260] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } }
I20240501 14:05:14.936582 6158 ts_tablet_manager.cc:1398] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:05:14.938287 6157 tablet_bootstrap.cc:492] T 71660575d94748d8a9329f909a42a0cd P 04088ee3bea440a7bd5ff772152b9410: No bootstrap required, opened a new log
I20240501 14:05:14.938498 6157 ts_tablet_manager.cc:1367] T 71660575d94748d8a9329f909a42a0cd P 04088ee3bea440a7bd5ff772152b9410: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.001s
I20240501 14:05:14.938769 6159 tablet_bootstrap.cc:492] T fee16905387d428aa02cd2070c86c790 P be56773105e744919601127a94e98adf: No bootstrap required, opened a new log
I20240501 14:05:14.938967 6159 ts_tablet_manager.cc:1367] T fee16905387d428aa02cd2070c86c790 P be56773105e744919601127a94e98adf: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.004s
I20240501 14:05:14.939321 6157 raft_consensus.cc:348] T 71660575d94748d8a9329f909a42a0cd P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:14.939565 6157 raft_consensus.cc:374] T 71660575d94748d8a9329f909a42a0cd P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:14.939677 6157 raft_consensus.cc:729] T 71660575d94748d8a9329f909a42a0cd P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04088ee3bea440a7bd5ff772152b9410, State: Initialized, Role: FOLLOWER
I20240501 14:05:14.939906 6159 raft_consensus.cc:348] T fee16905387d428aa02cd2070c86c790 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } }
I20240501 14:05:14.939922 6157 consensus_queue.cc:260] T 71660575d94748d8a9329f909a42a0cd P 04088ee3bea440a7bd5ff772152b9410 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:14.940130 6159 raft_consensus.cc:374] T fee16905387d428aa02cd2070c86c790 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:14.940209 6159 raft_consensus.cc:729] T fee16905387d428aa02cd2070c86c790 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be56773105e744919601127a94e98adf, State: Initialized, Role: FOLLOWER
I20240501 14:05:14.940450 6159 consensus_queue.cc:260] T fee16905387d428aa02cd2070c86c790 P be56773105e744919601127a94e98adf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } }
I20240501 14:05:14.940964 6157 ts_tablet_manager.cc:1398] T 71660575d94748d8a9329f909a42a0cd P 04088ee3bea440a7bd5ff772152b9410: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:05:14.941249 6157 tablet_bootstrap.cc:492] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410: Bootstrap starting.
I20240501 14:05:14.941565 6159 ts_tablet_manager.cc:1398] T fee16905387d428aa02cd2070c86c790 P be56773105e744919601127a94e98adf: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:05:14.941843 6159 tablet_bootstrap.cc:492] T 71660575d94748d8a9329f909a42a0cd P be56773105e744919601127a94e98adf: Bootstrap starting.
I20240501 14:05:14.942829 6157 tablet_bootstrap.cc:654] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:14.944346 6159 tablet_bootstrap.cc:654] T 71660575d94748d8a9329f909a42a0cd P be56773105e744919601127a94e98adf: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:14.945128 6157 tablet_bootstrap.cc:492] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410: No bootstrap required, opened a new log
I20240501 14:05:14.945317 6157 ts_tablet_manager.cc:1367] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240501 14:05:14.945981 6157 raft_consensus.cc:348] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } }
I20240501 14:05:14.946164 6159 tablet_bootstrap.cc:492] T 71660575d94748d8a9329f909a42a0cd P be56773105e744919601127a94e98adf: No bootstrap required, opened a new log
I20240501 14:05:14.946192 6157 raft_consensus.cc:374] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:14.946286 6159 ts_tablet_manager.cc:1367] T 71660575d94748d8a9329f909a42a0cd P be56773105e744919601127a94e98adf: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20240501 14:05:14.946324 6157 raft_consensus.cc:729] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04088ee3bea440a7bd5ff772152b9410, State: Initialized, Role: FOLLOWER
I20240501 14:05:14.946588 6157 consensus_queue.cc:260] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } }
I20240501 14:05:14.947114 6157 ts_tablet_manager.cc:1398] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:05:14.947048 6159 raft_consensus.cc:348] T 71660575d94748d8a9329f909a42a0cd P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:14.947274 6159 raft_consensus.cc:374] T 71660575d94748d8a9329f909a42a0cd P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:14.947384 6159 raft_consensus.cc:729] T 71660575d94748d8a9329f909a42a0cd P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be56773105e744919601127a94e98adf, State: Initialized, Role: FOLLOWER
I20240501 14:05:14.947623 6159 consensus_queue.cc:260] T 71660575d94748d8a9329f909a42a0cd P be56773105e744919601127a94e98adf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:14.948104 6159 ts_tablet_manager.cc:1398] T 71660575d94748d8a9329f909a42a0cd P be56773105e744919601127a94e98adf: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:05:15.063812 6138 raft_consensus.cc:482] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:05:15.064028 6138 raft_consensus.cc:504] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:15.064848 6138 leader_election.cc:288] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129:44033), be56773105e744919601127a94e98adf (127.29.124.132:35459)
I20240501 14:05:15.066110 5385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71660575d94748d8a9329f909a42a0cd" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf" is_pre_election: true
I20240501 14:05:15.066318 5385 raft_consensus.cc:2454] T 71660575d94748d8a9329f909a42a0cd P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 0.
I20240501 14:05:15.066859 5190 leader_election.cc:302] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters:
I20240501 14:05:15.067160 6138 raft_consensus.cc:2781] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:05:15.067265 6138 raft_consensus.cc:482] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:05:15.067356 6138 raft_consensus.cc:3037] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:15.070047 6138 raft_consensus.cc:504] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:15.070694 6138 leader_election.cc:288] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 1 election: Requested vote from peers 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129:44033), be56773105e744919601127a94e98adf (127.29.124.132:35459)
I20240501 14:05:15.071259 5385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71660575d94748d8a9329f909a42a0cd" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf"
I20240501 14:05:15.071430 5385 raft_consensus.cc:3037] T 71660575d94748d8a9329f909a42a0cd P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:15.071897 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71660575d94748d8a9329f909a42a0cd" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04088ee3bea440a7bd5ff772152b9410" is_pre_election: true
I20240501 14:05:15.071943 4992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71660575d94748d8a9329f909a42a0cd" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04088ee3bea440a7bd5ff772152b9410"
I20240501 14:05:15.072136 4995 raft_consensus.cc:2454] T 71660575d94748d8a9329f909a42a0cd P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 0.
I20240501 14:05:15.073534 5385 raft_consensus.cc:2454] T 71660575d94748d8a9329f909a42a0cd P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 1.
I20240501 14:05:15.073899 5190 leader_election.cc:302] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 04088ee3bea440a7bd5ff772152b9410
I20240501 14:05:15.074246 6138 raft_consensus.cc:2781] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:05:15.074393 6138 raft_consensus.cc:686] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEADER]: Becoming Leader. State: Replica: 1c3fdc5272484ac09ef3e5e36a08f27c, State: Running, Role: LEADER
I20240501 14:05:15.074657 6138 consensus_queue.cc:237] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:15.077415 4852 catalog_manager.cc:5484] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131). New cstate: current_term: 1 leader_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } health_report { overall_health: HEALTHY } } }
I20240501 14:05:15.127306 6166 raft_consensus.cc:482] T a726769d24284fedadd8e9e63cd27d05 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:05:15.127508 6166 raft_consensus.cc:504] T a726769d24284fedadd8e9e63cd27d05 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:15.129011 6166 leader_election.cc:288] T a726769d24284fedadd8e9e63cd27d05 P 04088ee3bea440a7bd5ff772152b9410 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377), be56773105e744919601127a94e98adf (127.29.124.132:35459)
I20240501 14:05:15.135583 5126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a726769d24284fedadd8e9e63cd27d05" candidate_uuid: "04088ee3bea440a7bd5ff772152b9410" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" is_pre_election: true
I20240501 14:05:15.135807 5126 raft_consensus.cc:2454] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04088ee3bea440a7bd5ff772152b9410 in term 0.
I20240501 14:05:15.136365 4929 leader_election.cc:302] T a726769d24284fedadd8e9e63cd27d05 P 04088ee3bea440a7bd5ff772152b9410 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04088ee3bea440a7bd5ff772152b9410, e9c60449cf3c46f8aca4ab6c535302a4; no voters:
I20240501 14:05:15.136632 5385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a726769d24284fedadd8e9e63cd27d05" candidate_uuid: "04088ee3bea440a7bd5ff772152b9410" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf" is_pre_election: true
I20240501 14:05:15.136790 6166 raft_consensus.cc:2781] T a726769d24284fedadd8e9e63cd27d05 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:05:15.136906 6166 raft_consensus.cc:482] T a726769d24284fedadd8e9e63cd27d05 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:05:15.136910 5385 raft_consensus.cc:2454] T a726769d24284fedadd8e9e63cd27d05 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04088ee3bea440a7bd5ff772152b9410 in term 0.
I20240501 14:05:15.136999 6166 raft_consensus.cc:3037] T a726769d24284fedadd8e9e63cd27d05 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:15.139087 6166 raft_consensus.cc:504] T a726769d24284fedadd8e9e63cd27d05 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:15.139649 6166 leader_election.cc:288] T a726769d24284fedadd8e9e63cd27d05 P 04088ee3bea440a7bd5ff772152b9410 [CANDIDATE]: Term 1 election: Requested vote from peers e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377), be56773105e744919601127a94e98adf (127.29.124.132:35459)
I20240501 14:05:15.140123 5126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a726769d24284fedadd8e9e63cd27d05" candidate_uuid: "04088ee3bea440a7bd5ff772152b9410" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9c60449cf3c46f8aca4ab6c535302a4"
I20240501 14:05:15.140169 5385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a726769d24284fedadd8e9e63cd27d05" candidate_uuid: "04088ee3bea440a7bd5ff772152b9410" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf"
I20240501 14:05:15.140311 5126 raft_consensus.cc:3037] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:15.140363 5385 raft_consensus.cc:3037] T a726769d24284fedadd8e9e63cd27d05 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:15.142243 5385 raft_consensus.cc:2454] T a726769d24284fedadd8e9e63cd27d05 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04088ee3bea440a7bd5ff772152b9410 in term 1.
I20240501 14:05:15.142263 5126 raft_consensus.cc:2454] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04088ee3bea440a7bd5ff772152b9410 in term 1.
I20240501 14:05:15.142637 4929 leader_election.cc:302] T a726769d24284fedadd8e9e63cd27d05 P 04088ee3bea440a7bd5ff772152b9410 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04088ee3bea440a7bd5ff772152b9410, be56773105e744919601127a94e98adf; no voters:
I20240501 14:05:15.142925 6166 raft_consensus.cc:2781] T a726769d24284fedadd8e9e63cd27d05 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:05:15.143318 6166 raft_consensus.cc:686] T a726769d24284fedadd8e9e63cd27d05 P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEADER]: Becoming Leader. State: Replica: 04088ee3bea440a7bd5ff772152b9410, State: Running, Role: LEADER
I20240501 14:05:15.143784 6166 consensus_queue.cc:237] T a726769d24284fedadd8e9e63cd27d05 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:15.147087 4852 catalog_manager.cc:5484] T a726769d24284fedadd8e9e63cd27d05 P 04088ee3bea440a7bd5ff772152b9410 reported cstate change: term changed from 0 to 1, leader changed from <none> to 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129). New cstate: current_term: 1 leader_uuid: "04088ee3bea440a7bd5ff772152b9410" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } health_report { overall_health: UNKNOWN } } }
I20240501 14:05:15.251806 6166 raft_consensus.cc:482] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:05:15.252036 6166 raft_consensus.cc:504] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } }
I20240501 14:05:15.252719 6166 leader_election.cc:288] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377), be56773105e744919601127a94e98adf (127.29.124.132:35459)
I20240501 14:05:15.253244 5126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fee16905387d428aa02cd2070c86c790" candidate_uuid: "04088ee3bea440a7bd5ff772152b9410" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" is_pre_election: true
I20240501 14:05:15.253355 5385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fee16905387d428aa02cd2070c86c790" candidate_uuid: "04088ee3bea440a7bd5ff772152b9410" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf" is_pre_election: true
I20240501 14:05:15.253544 5126 raft_consensus.cc:2454] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04088ee3bea440a7bd5ff772152b9410 in term 0.
I20240501 14:05:15.253612 5385 raft_consensus.cc:2454] T fee16905387d428aa02cd2070c86c790 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04088ee3bea440a7bd5ff772152b9410 in term 0.
I20240501 14:05:15.253994 4929 leader_election.cc:302] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04088ee3bea440a7bd5ff772152b9410, e9c60449cf3c46f8aca4ab6c535302a4; no voters:
I20240501 14:05:15.254292 6166 raft_consensus.cc:2781] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:05:15.254391 6166 raft_consensus.cc:482] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:05:15.254489 6166 raft_consensus.cc:3037] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:15.256402 6166 raft_consensus.cc:504] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } }
I20240501 14:05:15.256965 6166 leader_election.cc:288] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [CANDIDATE]: Term 1 election: Requested vote from peers e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377), be56773105e744919601127a94e98adf (127.29.124.132:35459)
I20240501 14:05:15.257427 5126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fee16905387d428aa02cd2070c86c790" candidate_uuid: "04088ee3bea440a7bd5ff772152b9410" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9c60449cf3c46f8aca4ab6c535302a4"
I20240501 14:05:15.257485 5385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fee16905387d428aa02cd2070c86c790" candidate_uuid: "04088ee3bea440a7bd5ff772152b9410" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf"
I20240501 14:05:15.257648 5126 raft_consensus.cc:3037] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:15.257668 5385 raft_consensus.cc:3037] T fee16905387d428aa02cd2070c86c790 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:15.260332 5385 raft_consensus.cc:2454] T fee16905387d428aa02cd2070c86c790 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04088ee3bea440a7bd5ff772152b9410 in term 1.
I20240501 14:05:15.260329 5126 raft_consensus.cc:2454] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04088ee3bea440a7bd5ff772152b9410 in term 1.
I20240501 14:05:15.260713 4929 leader_election.cc:302] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04088ee3bea440a7bd5ff772152b9410, e9c60449cf3c46f8aca4ab6c535302a4; no voters:
I20240501 14:05:15.260979 6166 raft_consensus.cc:2781] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:05:15.261155 6166 raft_consensus.cc:686] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEADER]: Becoming Leader. State: Replica: 04088ee3bea440a7bd5ff772152b9410, State: Running, Role: LEADER
I20240501 14:05:15.261416 6166 consensus_queue.cc:237] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } }
I20240501 14:05:15.264509 4852 catalog_manager.cc:5484] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 reported cstate change: term changed from 0 to 1, leader changed from <none> to 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129). New cstate: current_term: 1 leader_uuid: "04088ee3bea440a7bd5ff772152b9410" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } health_report { overall_health: HEALTHY } } }
I20240501 14:05:15.512535 4992 raft_consensus.cc:3037] T 71660575d94748d8a9329f909a42a0cd P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:15.516207 6138 consensus_queue.cc:1035] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:05:15.522312 6138 consensus_queue.cc:1035] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:05:15.614646 6166 consensus_queue.cc:1035] T a726769d24284fedadd8e9e63cd27d05 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:05:15.621335 6166 consensus_queue.cc:1035] T a726769d24284fedadd8e9e63cd27d05 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:05:15.626533 6179 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:05:15.626930 6179 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:05:15.634328 6179 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240501 14:05:15.729869 6166 consensus_queue.cc:1035] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:05:15.737032 6176 consensus_queue.cc:1035] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 214.37 ms
time per row: 0.107185 ms
I20240501 14:05:15.987269 4847 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55620:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:05:15.996246 4974 tablet_service.cc:1467] Processing CreateTablet for tablet 9afe3b20a43d4b5bb800af39f945948d (DEFAULT_TABLE table=rebalance_test_table_3 [id=41272b857ab045629ca60ab2467097b0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:15.996249 5236 tablet_service.cc:1467] Processing CreateTablet for tablet 9afe3b20a43d4b5bb800af39f945948d (DEFAULT_TABLE table=rebalance_test_table_3 [id=41272b857ab045629ca60ab2467097b0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:15.996347 5105 tablet_service.cc:1467] Processing CreateTablet for tablet 9afe3b20a43d4b5bb800af39f945948d (DEFAULT_TABLE table=rebalance_test_table_3 [id=41272b857ab045629ca60ab2467097b0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:15.996711 4974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9afe3b20a43d4b5bb800af39f945948d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:15.996712 5236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9afe3b20a43d4b5bb800af39f945948d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:15.996721 5105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9afe3b20a43d4b5bb800af39f945948d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:15.997890 4973 tablet_service.cc:1467] Processing CreateTablet for tablet 51cbcf4888c440c2ad096c988ab2d968 (DEFAULT_TABLE table=rebalance_test_table_3 [id=41272b857ab045629ca60ab2467097b0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:15.997958 5366 tablet_service.cc:1467] Processing CreateTablet for tablet 51cbcf4888c440c2ad096c988ab2d968 (DEFAULT_TABLE table=rebalance_test_table_3 [id=41272b857ab045629ca60ab2467097b0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:15.998281 4973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51cbcf4888c440c2ad096c988ab2d968. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:15.998308 5366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51cbcf4888c440c2ad096c988ab2d968. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:16.001627 5365 tablet_service.cc:1467] Processing CreateTablet for tablet 404e2851055541f0be1bb0971a7cc72a (DEFAULT_TABLE table=rebalance_test_table_3 [id=41272b857ab045629ca60ab2467097b0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:16.001996 5365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 404e2851055541f0be1bb0971a7cc72a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:16.003342 5106 tablet_service.cc:1467] Processing CreateTablet for tablet 404e2851055541f0be1bb0971a7cc72a (DEFAULT_TABLE table=rebalance_test_table_3 [id=41272b857ab045629ca60ab2467097b0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:16.003782 5106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 404e2851055541f0be1bb0971a7cc72a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:16.006175 5235 tablet_service.cc:1467] Processing CreateTablet for tablet 51cbcf4888c440c2ad096c988ab2d968 (DEFAULT_TABLE table=rebalance_test_table_3 [id=41272b857ab045629ca60ab2467097b0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:16.006594 5235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51cbcf4888c440c2ad096c988ab2d968. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:16.007486 6241 tablet_bootstrap.cc:492] T 9afe3b20a43d4b5bb800af39f945948d P 04088ee3bea440a7bd5ff772152b9410: Bootstrap starting.
I20240501 14:05:16.009223 6241 tablet_bootstrap.cc:654] T 9afe3b20a43d4b5bb800af39f945948d P 04088ee3bea440a7bd5ff772152b9410: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:16.009660 5234 tablet_service.cc:1467] Processing CreateTablet for tablet 404e2851055541f0be1bb0971a7cc72a (DEFAULT_TABLE table=rebalance_test_table_3 [id=41272b857ab045629ca60ab2467097b0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:16.010066 5234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 404e2851055541f0be1bb0971a7cc72a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:16.013440 6242 tablet_bootstrap.cc:492] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf: Bootstrap starting.
I20240501 14:05:16.014196 6241 tablet_bootstrap.cc:492] T 9afe3b20a43d4b5bb800af39f945948d P 04088ee3bea440a7bd5ff772152b9410: No bootstrap required, opened a new log
I20240501 14:05:16.014331 6241 ts_tablet_manager.cc:1367] T 9afe3b20a43d4b5bb800af39f945948d P 04088ee3bea440a7bd5ff772152b9410: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20240501 14:05:16.014971 6241 raft_consensus.cc:348] T 9afe3b20a43d4b5bb800af39f945948d P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } }
I20240501 14:05:16.015146 6241 raft_consensus.cc:374] T 9afe3b20a43d4b5bb800af39f945948d P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:16.015378 6242 tablet_bootstrap.cc:654] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:16.016625 6241 raft_consensus.cc:729] T 9afe3b20a43d4b5bb800af39f945948d P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04088ee3bea440a7bd5ff772152b9410, State: Initialized, Role: FOLLOWER
I20240501 14:05:16.016883 6241 consensus_queue.cc:260] T 9afe3b20a43d4b5bb800af39f945948d P 04088ee3bea440a7bd5ff772152b9410 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } }
I20240501 14:05:16.017412 6242 tablet_bootstrap.cc:492] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf: No bootstrap required, opened a new log
I20240501 14:05:16.017426 6241 ts_tablet_manager.cc:1398] T 9afe3b20a43d4b5bb800af39f945948d P 04088ee3bea440a7bd5ff772152b9410: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20240501 14:05:16.017590 6242 ts_tablet_manager.cc:1367] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.003s
I20240501 14:05:16.018549 6242 raft_consensus.cc:348] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:16.018747 6242 raft_consensus.cc:374] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:16.018882 6242 raft_consensus.cc:729] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be56773105e744919601127a94e98adf, State: Initialized, Role: FOLLOWER
I20240501 14:05:16.019125 6242 consensus_queue.cc:260] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:16.019687 6242 ts_tablet_manager.cc:1398] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:05:16.025609 6241 tablet_bootstrap.cc:492] T 51cbcf4888c440c2ad096c988ab2d968 P 04088ee3bea440a7bd5ff772152b9410: Bootstrap starting.
I20240501 14:05:16.027171 6241 tablet_bootstrap.cc:654] T 51cbcf4888c440c2ad096c988ab2d968 P 04088ee3bea440a7bd5ff772152b9410: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:16.028225 6242 tablet_bootstrap.cc:492] T 404e2851055541f0be1bb0971a7cc72a P be56773105e744919601127a94e98adf: Bootstrap starting.
I20240501 14:05:16.029773 6242 tablet_bootstrap.cc:654] T 404e2851055541f0be1bb0971a7cc72a P be56773105e744919601127a94e98adf: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:16.030370 6240 tablet_bootstrap.cc:492] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4: Bootstrap starting.
I20240501 14:05:16.032066 6240 tablet_bootstrap.cc:654] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:16.033716 6246 tablet_bootstrap.cc:492] T 9afe3b20a43d4b5bb800af39f945948d P 1c3fdc5272484ac09ef3e5e36a08f27c: Bootstrap starting.
I20240501 14:05:16.033922 6241 tablet_bootstrap.cc:492] T 51cbcf4888c440c2ad096c988ab2d968 P 04088ee3bea440a7bd5ff772152b9410: No bootstrap required, opened a new log
I20240501 14:05:16.034117 6241 ts_tablet_manager.cc:1367] T 51cbcf4888c440c2ad096c988ab2d968 P 04088ee3bea440a7bd5ff772152b9410: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20240501 14:05:16.034157 6240 tablet_bootstrap.cc:492] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4: No bootstrap required, opened a new log
I20240501 14:05:16.034279 6240 ts_tablet_manager.cc:1367] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:05:16.034884 6241 raft_consensus.cc:348] T 51cbcf4888c440c2ad096c988ab2d968 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:16.035037 6246 tablet_bootstrap.cc:654] T 9afe3b20a43d4b5bb800af39f945948d P 1c3fdc5272484ac09ef3e5e36a08f27c: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:16.034991 6240 raft_consensus.cc:348] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } }
I20240501 14:05:16.035120 6241 raft_consensus.cc:374] T 51cbcf4888c440c2ad096c988ab2d968 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:16.035192 6240 raft_consensus.cc:374] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:16.035187 6241 raft_consensus.cc:729] T 51cbcf4888c440c2ad096c988ab2d968 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04088ee3bea440a7bd5ff772152b9410, State: Initialized, Role: FOLLOWER
I20240501 14:05:16.035270 6240 raft_consensus.cc:729] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9c60449cf3c46f8aca4ab6c535302a4, State: Initialized, Role: FOLLOWER
I20240501 14:05:16.035446 6241 consensus_queue.cc:260] T 51cbcf4888c440c2ad096c988ab2d968 P 04088ee3bea440a7bd5ff772152b9410 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:16.035468 6240 consensus_queue.cc:260] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } }
I20240501 14:05:16.035967 6240 ts_tablet_manager.cc:1398] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:05:16.036254 6240 tablet_bootstrap.cc:492] T 404e2851055541f0be1bb0971a7cc72a P e9c60449cf3c46f8aca4ab6c535302a4: Bootstrap starting.
I20240501 14:05:16.036561 6241 ts_tablet_manager.cc:1398] T 51cbcf4888c440c2ad096c988ab2d968 P 04088ee3bea440a7bd5ff772152b9410: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240501 14:05:16.038995 6240 tablet_bootstrap.cc:654] T 404e2851055541f0be1bb0971a7cc72a P e9c60449cf3c46f8aca4ab6c535302a4: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:16.040390 6242 tablet_bootstrap.cc:492] T 404e2851055541f0be1bb0971a7cc72a P be56773105e744919601127a94e98adf: No bootstrap required, opened a new log
I20240501 14:05:16.040851 6242 ts_tablet_manager.cc:1367] T 404e2851055541f0be1bb0971a7cc72a P be56773105e744919601127a94e98adf: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.003s
I20240501 14:05:16.041711 6242 raft_consensus.cc:348] T 404e2851055541f0be1bb0971a7cc72a P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:16.042023 6242 raft_consensus.cc:374] T 404e2851055541f0be1bb0971a7cc72a P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:16.042143 6242 raft_consensus.cc:729] T 404e2851055541f0be1bb0971a7cc72a P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be56773105e744919601127a94e98adf, State: Initialized, Role: FOLLOWER
I20240501 14:05:16.042484 6242 consensus_queue.cc:260] T 404e2851055541f0be1bb0971a7cc72a P be56773105e744919601127a94e98adf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:16.043082 6242 ts_tablet_manager.cc:1398] T 404e2851055541f0be1bb0971a7cc72a P be56773105e744919601127a94e98adf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:05:16.043455 6240 tablet_bootstrap.cc:492] T 404e2851055541f0be1bb0971a7cc72a P e9c60449cf3c46f8aca4ab6c535302a4: No bootstrap required, opened a new log
I20240501 14:05:16.043610 6240 ts_tablet_manager.cc:1367] T 404e2851055541f0be1bb0971a7cc72a P e9c60449cf3c46f8aca4ab6c535302a4: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20240501 14:05:16.043692 6246 tablet_bootstrap.cc:492] T 9afe3b20a43d4b5bb800af39f945948d P 1c3fdc5272484ac09ef3e5e36a08f27c: No bootstrap required, opened a new log
I20240501 14:05:16.043892 6246 ts_tablet_manager.cc:1367] T 9afe3b20a43d4b5bb800af39f945948d P 1c3fdc5272484ac09ef3e5e36a08f27c: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20240501 14:05:16.044401 6240 raft_consensus.cc:348] T 404e2851055541f0be1bb0971a7cc72a P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:16.044660 6240 raft_consensus.cc:374] T 404e2851055541f0be1bb0971a7cc72a P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:16.044747 6240 raft_consensus.cc:729] T 404e2851055541f0be1bb0971a7cc72a P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9c60449cf3c46f8aca4ab6c535302a4, State: Initialized, Role: FOLLOWER
I20240501 14:05:16.044757 6246 raft_consensus.cc:348] T 9afe3b20a43d4b5bb800af39f945948d P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } }
I20240501 14:05:16.044945 6246 raft_consensus.cc:374] T 9afe3b20a43d4b5bb800af39f945948d P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:16.045046 6246 raft_consensus.cc:729] T 9afe3b20a43d4b5bb800af39f945948d P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c3fdc5272484ac09ef3e5e36a08f27c, State: Initialized, Role: FOLLOWER
I20240501 14:05:16.045004 6240 consensus_queue.cc:260] T 404e2851055541f0be1bb0971a7cc72a P e9c60449cf3c46f8aca4ab6c535302a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:16.045611 6240 ts_tablet_manager.cc:1398] T 404e2851055541f0be1bb0971a7cc72a P e9c60449cf3c46f8aca4ab6c535302a4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:05:16.045987 6246 consensus_queue.cc:260] T 9afe3b20a43d4b5bb800af39f945948d P 1c3fdc5272484ac09ef3e5e36a08f27c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } }
I20240501 14:05:16.046523 6246 ts_tablet_manager.cc:1398] T 9afe3b20a43d4b5bb800af39f945948d P 1c3fdc5272484ac09ef3e5e36a08f27c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:05:16.046869 6246 tablet_bootstrap.cc:492] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c: Bootstrap starting.
I20240501 14:05:16.048358 6246 tablet_bootstrap.cc:654] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:16.050076 6246 tablet_bootstrap.cc:492] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c: No bootstrap required, opened a new log
I20240501 14:05:16.050226 6246 ts_tablet_manager.cc:1367] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240501 14:05:16.050887 6246 raft_consensus.cc:348] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:16.051111 6246 raft_consensus.cc:374] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:16.051229 6246 raft_consensus.cc:729] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c3fdc5272484ac09ef3e5e36a08f27c, State: Initialized, Role: FOLLOWER
I20240501 14:05:16.051486 6246 consensus_queue.cc:260] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:16.052047 6246 ts_tablet_manager.cc:1398] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:05:16.052309 6246 tablet_bootstrap.cc:492] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c: Bootstrap starting.
I20240501 14:05:16.053787 6246 tablet_bootstrap.cc:654] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:16.055467 6246 tablet_bootstrap.cc:492] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c: No bootstrap required, opened a new log
I20240501 14:05:16.055572 6246 ts_tablet_manager.cc:1367] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20240501 14:05:16.056084 6246 raft_consensus.cc:348] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:16.056222 6246 raft_consensus.cc:374] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:16.056285 6246 raft_consensus.cc:729] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c3fdc5272484ac09ef3e5e36a08f27c, State: Initialized, Role: FOLLOWER
I20240501 14:05:16.056455 6246 consensus_queue.cc:260] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:16.056813 6246 ts_tablet_manager.cc:1398] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:05:16.245622 6225 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:05:16.245815 6225 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:16.246526 6225 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be56773105e744919601127a94e98adf (127.29.124.132:35459), 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129:44033)
I20240501 14:05:16.247156 5384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf" is_pre_election: true
I20240501 14:05:16.247161 4992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04088ee3bea440a7bd5ff772152b9410" is_pre_election: true
I20240501 14:05:16.247433 5384 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 0.
I20240501 14:05:16.247432 4992 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 0.
I20240501 14:05:16.247888 5190 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04088ee3bea440a7bd5ff772152b9410, 1c3fdc5272484ac09ef3e5e36a08f27c; no voters:
I20240501 14:05:16.248170 6225 raft_consensus.cc:2781] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:05:16.248291 6225 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:05:16.248411 6225 raft_consensus.cc:3037] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:16.249372 6138 raft_consensus.cc:482] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:05:16.249534 6138 raft_consensus.cc:504] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:16.250046 6138 leader_election.cc:288] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be56773105e744919601127a94e98adf (127.29.124.132:35459), e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377)
I20240501 14:05:16.250610 5126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "404e2851055541f0be1bb0971a7cc72a" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" is_pre_election: true
I20240501 14:05:16.250809 5126 raft_consensus.cc:2454] T 404e2851055541f0be1bb0971a7cc72a P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 0.
I20240501 14:05:16.250782 6225 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:16.251233 5190 leader_election.cc:302] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, e9c60449cf3c46f8aca4ab6c535302a4; no voters:
I20240501 14:05:16.251245 5384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "404e2851055541f0be1bb0971a7cc72a" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf" is_pre_election: true
I20240501 14:05:16.251278 6225 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 1 election: Requested vote from peers be56773105e744919601127a94e98adf (127.29.124.132:35459), 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129:44033)
I20240501 14:05:16.251426 5384 raft_consensus.cc:2454] T 404e2851055541f0be1bb0971a7cc72a P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 0.
I20240501 14:05:16.251531 6225 raft_consensus.cc:2781] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:05:16.251629 6225 raft_consensus.cc:482] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:05:16.251725 6225 raft_consensus.cc:3037] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:16.251883 5384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf"
I20240501 14:05:16.252060 5384 raft_consensus.cc:3037] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:16.252081 4992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04088ee3bea440a7bd5ff772152b9410"
I20240501 14:05:16.252277 4992 raft_consensus.cc:3037] T 51cbcf4888c440c2ad096c988ab2d968 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:16.254005 5384 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 1.
I20240501 14:05:16.254220 6225 raft_consensus.cc:504] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:16.254540 5190 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters:
I20240501 14:05:16.254817 6138 raft_consensus.cc:2781] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:05:16.254969 6138 raft_consensus.cc:686] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEADER]: Becoming Leader. State: Replica: 1c3fdc5272484ac09ef3e5e36a08f27c, State: Running, Role: LEADER
I20240501 14:05:16.254890 6225 leader_election.cc:288] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 1 election: Requested vote from peers be56773105e744919601127a94e98adf (127.29.124.132:35459), e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377)
I20240501 14:05:16.255263 6138 consensus_queue.cc:237] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:16.255424 5126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "404e2851055541f0be1bb0971a7cc72a" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9c60449cf3c46f8aca4ab6c535302a4"
I20240501 14:05:16.255559 5384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "404e2851055541f0be1bb0971a7cc72a" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf"
I20240501 14:05:16.255609 5126 raft_consensus.cc:3037] T 404e2851055541f0be1bb0971a7cc72a P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:16.255839 4992 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 1.
I20240501 14:05:16.256104 5384 raft_consensus.cc:3037] T 404e2851055541f0be1bb0971a7cc72a P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:16.257586 5126 raft_consensus.cc:2454] T 404e2851055541f0be1bb0971a7cc72a P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 1.
I20240501 14:05:16.258021 5190 leader_election.cc:302] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, e9c60449cf3c46f8aca4ab6c535302a4; no voters:
I20240501 14:05:16.258297 6138 raft_consensus.cc:2781] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:05:16.258553 6138 raft_consensus.cc:686] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEADER]: Becoming Leader. State: Replica: 1c3fdc5272484ac09ef3e5e36a08f27c, State: Running, Role: LEADER
I20240501 14:05:16.258752 5384 raft_consensus.cc:2454] T 404e2851055541f0be1bb0971a7cc72a P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 1.
I20240501 14:05:16.258790 6138 consensus_queue.cc:237] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:16.261960 4851 catalog_manager.cc:5484] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131). New cstate: current_term: 1 leader_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } health_report { overall_health: HEALTHY } } }
I20240501 14:05:16.262521 4851 catalog_manager.cc:5484] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131). New cstate: current_term: 1 leader_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } health_report { overall_health: HEALTHY } } }
I20240501 14:05:16.350409 5999 raft_consensus.cc:482] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:05:16.350636 5999 raft_consensus.cc:504] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } }
I20240501 14:05:16.351338 5999 leader_election.cc:288] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209), 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129:44033)
I20240501 14:05:16.351748 5254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9afe3b20a43d4b5bb800af39f945948d" candidate_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" is_pre_election: true
I20240501 14:05:16.351930 4992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9afe3b20a43d4b5bb800af39f945948d" candidate_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04088ee3bea440a7bd5ff772152b9410" is_pre_election: true
I20240501 14:05:16.351996 5254 raft_consensus.cc:2454] T 9afe3b20a43d4b5bb800af39f945948d P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9c60449cf3c46f8aca4ab6c535302a4 in term 0.
I20240501 14:05:16.352160 4992 raft_consensus.cc:2454] T 9afe3b20a43d4b5bb800af39f945948d P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9c60449cf3c46f8aca4ab6c535302a4 in term 0.
I20240501 14:05:16.352444 5059 leader_election.cc:302] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, e9c60449cf3c46f8aca4ab6c535302a4; no voters:
I20240501 14:05:16.352725 5999 raft_consensus.cc:2781] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:05:16.352810 5999 raft_consensus.cc:482] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:05:16.352910 5999 raft_consensus.cc:3037] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:16.354768 5999 raft_consensus.cc:504] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } }
I20240501 14:05:16.355319 5999 leader_election.cc:288] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 [CANDIDATE]: Term 1 election: Requested vote from peers 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209), 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129:44033)
I20240501 14:05:16.355741 5254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9afe3b20a43d4b5bb800af39f945948d" candidate_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c"
I20240501 14:05:16.355741 4992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9afe3b20a43d4b5bb800af39f945948d" candidate_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04088ee3bea440a7bd5ff772152b9410"
I20240501 14:05:16.355927 5254 raft_consensus.cc:3037] T 9afe3b20a43d4b5bb800af39f945948d P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:16.355947 4992 raft_consensus.cc:3037] T 9afe3b20a43d4b5bb800af39f945948d P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:16.358608 5254 raft_consensus.cc:2454] T 9afe3b20a43d4b5bb800af39f945948d P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9c60449cf3c46f8aca4ab6c535302a4 in term 1.
I20240501 14:05:16.358595 4992 raft_consensus.cc:2454] T 9afe3b20a43d4b5bb800af39f945948d P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9c60449cf3c46f8aca4ab6c535302a4 in term 1.
I20240501 14:05:16.358950 5059 leader_election.cc:302] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, e9c60449cf3c46f8aca4ab6c535302a4; no voters:
I20240501 14:05:16.359215 5999 raft_consensus.cc:2781] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:05:16.359558 5999 raft_consensus.cc:686] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 LEADER]: Becoming Leader. State: Replica: e9c60449cf3c46f8aca4ab6c535302a4, State: Running, Role: LEADER
I20240501 14:05:16.359812 5999 consensus_queue.cc:237] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } }
I20240501 14:05:16.362318 4851 catalog_manager.cc:5484] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130). New cstate: current_term: 1 leader_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: HEALTHY } } }
I20240501 14:05:16.650314 6138 consensus_queue.cc:1035] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:05:16.683435 6138 consensus_queue.cc:1035] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:05:16.728219 6257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:05:16.728556 6257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 14:05:16.731451 6138 consensus_queue.cc:1035] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:05:16.736896 5992 consensus_queue.cc:1035] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:05:16.738296 6257 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
I20240501 14:05:16.839653 6255 consensus_queue.cc:1035] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:05:16.873884 6255 consensus_queue.cc:1035] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 214.948 ms
time per row: 0.107474 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240501 14:05:17.959553 4847 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55620:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:05:17.967967 4973 tablet_service.cc:1467] Processing CreateTablet for tablet ba6cba753e934e719c730593c0706346 (DEFAULT_TABLE table=rebalance_test_table_4 [id=84aeba643ddd44cfbbd6a34f81703482]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:17.968456 4973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba6cba753e934e719c730593c0706346. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:17.968466 5105 tablet_service.cc:1467] Processing CreateTablet for tablet ba6cba753e934e719c730593c0706346 (DEFAULT_TABLE table=rebalance_test_table_4 [id=84aeba643ddd44cfbbd6a34f81703482]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:17.968883 5105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba6cba753e934e719c730593c0706346. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:17.969218 5234 tablet_service.cc:1467] Processing CreateTablet for tablet ba6cba753e934e719c730593c0706346 (DEFAULT_TABLE table=rebalance_test_table_4 [id=84aeba643ddd44cfbbd6a34f81703482]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:17.969671 5234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba6cba753e934e719c730593c0706346. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:17.972695 4974 tablet_service.cc:1467] Processing CreateTablet for tablet b284aa82594a44f5885f32fa05844913 (DEFAULT_TABLE table=rebalance_test_table_4 [id=84aeba643ddd44cfbbd6a34f81703482]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:17.973070 4974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b284aa82594a44f5885f32fa05844913. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:17.974490 5366 tablet_service.cc:1467] Processing CreateTablet for tablet e7d5f02ccba34254a4d20951878b1214 (DEFAULT_TABLE table=rebalance_test_table_4 [id=84aeba643ddd44cfbbd6a34f81703482]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:17.974922 5366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7d5f02ccba34254a4d20951878b1214. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:17.977921 5106 tablet_service.cc:1467] Processing CreateTablet for tablet b284aa82594a44f5885f32fa05844913 (DEFAULT_TABLE table=rebalance_test_table_4 [id=84aeba643ddd44cfbbd6a34f81703482]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:17.978322 5106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b284aa82594a44f5885f32fa05844913. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:17.979075 5365 tablet_service.cc:1467] Processing CreateTablet for tablet b284aa82594a44f5885f32fa05844913 (DEFAULT_TABLE table=rebalance_test_table_4 [id=84aeba643ddd44cfbbd6a34f81703482]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:17.979461 5365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b284aa82594a44f5885f32fa05844913. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:17.983008 6323 tablet_bootstrap.cc:492] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410: Bootstrap starting.
I20240501 14:05:17.982995 5236 tablet_service.cc:1467] Processing CreateTablet for tablet e7d5f02ccba34254a4d20951878b1214 (DEFAULT_TABLE table=rebalance_test_table_4 [id=84aeba643ddd44cfbbd6a34f81703482]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:17.983390 5236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7d5f02ccba34254a4d20951878b1214. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:17.984540 6323 tablet_bootstrap.cc:654] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:17.993027 6326 tablet_bootstrap.cc:492] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf: Bootstrap starting.
I20240501 14:05:17.995113 6326 tablet_bootstrap.cc:654] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:17.995877 4974 tablet_service.cc:1467] Processing CreateTablet for tablet e7d5f02ccba34254a4d20951878b1214 (DEFAULT_TABLE table=rebalance_test_table_4 [id=84aeba643ddd44cfbbd6a34f81703482]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:17.996302 4974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7d5f02ccba34254a4d20951878b1214. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:18.006048 6324 tablet_bootstrap.cc:492] T ba6cba753e934e719c730593c0706346 P e9c60449cf3c46f8aca4ab6c535302a4: Bootstrap starting.
I20240501 14:05:18.007462 6324 tablet_bootstrap.cc:654] T ba6cba753e934e719c730593c0706346 P e9c60449cf3c46f8aca4ab6c535302a4: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:18.011509 6327 tablet_bootstrap.cc:492] T e7d5f02ccba34254a4d20951878b1214 P 1c3fdc5272484ac09ef3e5e36a08f27c: Bootstrap starting.
I20240501 14:05:18.012326 6324 tablet_bootstrap.cc:492] T ba6cba753e934e719c730593c0706346 P e9c60449cf3c46f8aca4ab6c535302a4: No bootstrap required, opened a new log
I20240501 14:05:18.012467 6324 ts_tablet_manager.cc:1367] T ba6cba753e934e719c730593c0706346 P e9c60449cf3c46f8aca4ab6c535302a4: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:05:18.013005 6324 raft_consensus.cc:348] T ba6cba753e934e719c730593c0706346 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } }
I20240501 14:05:18.013165 6324 raft_consensus.cc:374] T ba6cba753e934e719c730593c0706346 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:18.013284 6324 raft_consensus.cc:729] T ba6cba753e934e719c730593c0706346 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9c60449cf3c46f8aca4ab6c535302a4, State: Initialized, Role: FOLLOWER
I20240501 14:05:18.013559 6327 tablet_bootstrap.cc:654] T e7d5f02ccba34254a4d20951878b1214 P 1c3fdc5272484ac09ef3e5e36a08f27c: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:18.013520 6324 consensus_queue.cc:260] T ba6cba753e934e719c730593c0706346 P e9c60449cf3c46f8aca4ab6c535302a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } }
I20240501 14:05:18.014122 6324 ts_tablet_manager.cc:1398] T ba6cba753e934e719c730593c0706346 P e9c60449cf3c46f8aca4ab6c535302a4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:05:18.014449 6324 tablet_bootstrap.cc:492] T b284aa82594a44f5885f32fa05844913 P e9c60449cf3c46f8aca4ab6c535302a4: Bootstrap starting.
I20240501 14:05:18.015441 6323 tablet_bootstrap.cc:492] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410: No bootstrap required, opened a new log
I20240501 14:05:18.015673 6323 ts_tablet_manager.cc:1367] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410: Time spent bootstrapping tablet: real 0.033s user 0.016s sys 0.016s
I20240501 14:05:18.015947 6324 tablet_bootstrap.cc:654] T b284aa82594a44f5885f32fa05844913 P e9c60449cf3c46f8aca4ab6c535302a4: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:18.016479 6323 raft_consensus.cc:348] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } }
I20240501 14:05:18.016691 6323 raft_consensus.cc:374] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:18.016781 6323 raft_consensus.cc:729] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04088ee3bea440a7bd5ff772152b9410, State: Initialized, Role: FOLLOWER
I20240501 14:05:18.017318 6323 consensus_queue.cc:260] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } }
I20240501 14:05:18.017776 6326 tablet_bootstrap.cc:492] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf: No bootstrap required, opened a new log
I20240501 14:05:18.017979 6326 ts_tablet_manager.cc:1367] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf: Time spent bootstrapping tablet: real 0.025s user 0.000s sys 0.007s
I20240501 14:05:18.018035 6323 ts_tablet_manager.cc:1398] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:05:18.018326 6323 tablet_bootstrap.cc:492] T b284aa82594a44f5885f32fa05844913 P 04088ee3bea440a7bd5ff772152b9410: Bootstrap starting.
I20240501 14:05:18.018827 6326 raft_consensus.cc:348] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:18.019060 6326 raft_consensus.cc:374] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:18.019199 6326 raft_consensus.cc:729] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be56773105e744919601127a94e98adf, State: Initialized, Role: FOLLOWER
I20240501 14:05:18.019517 6326 consensus_queue.cc:260] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:18.019825 6323 tablet_bootstrap.cc:654] T b284aa82594a44f5885f32fa05844913 P 04088ee3bea440a7bd5ff772152b9410: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:18.020134 6326 ts_tablet_manager.cc:1398] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:05:18.020426 6326 tablet_bootstrap.cc:492] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf: Bootstrap starting.
I20240501 14:05:18.021924 6326 tablet_bootstrap.cc:654] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:18.022658 6327 tablet_bootstrap.cc:492] T e7d5f02ccba34254a4d20951878b1214 P 1c3fdc5272484ac09ef3e5e36a08f27c: No bootstrap required, opened a new log
I20240501 14:05:18.022795 6327 ts_tablet_manager.cc:1367] T e7d5f02ccba34254a4d20951878b1214 P 1c3fdc5272484ac09ef3e5e36a08f27c: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20240501 14:05:18.023463 6323 tablet_bootstrap.cc:492] T b284aa82594a44f5885f32fa05844913 P 04088ee3bea440a7bd5ff772152b9410: No bootstrap required, opened a new log
I20240501 14:05:18.023653 6323 ts_tablet_manager.cc:1367] T b284aa82594a44f5885f32fa05844913 P 04088ee3bea440a7bd5ff772152b9410: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20240501 14:05:18.023913 6327 raft_consensus.cc:348] T e7d5f02ccba34254a4d20951878b1214 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } }
I20240501 14:05:18.024160 6327 raft_consensus.cc:374] T e7d5f02ccba34254a4d20951878b1214 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:18.024296 6327 raft_consensus.cc:729] T e7d5f02ccba34254a4d20951878b1214 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c3fdc5272484ac09ef3e5e36a08f27c, State: Initialized, Role: FOLLOWER
I20240501 14:05:18.024359 6323 raft_consensus.cc:348] T b284aa82594a44f5885f32fa05844913 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:18.024574 6323 raft_consensus.cc:374] T b284aa82594a44f5885f32fa05844913 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:18.024657 6323 raft_consensus.cc:729] T b284aa82594a44f5885f32fa05844913 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04088ee3bea440a7bd5ff772152b9410, State: Initialized, Role: FOLLOWER
I20240501 14:05:18.024624 6327 consensus_queue.cc:260] T e7d5f02ccba34254a4d20951878b1214 P 1c3fdc5272484ac09ef3e5e36a08f27c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } }
I20240501 14:05:18.024879 6323 consensus_queue.cc:260] T b284aa82594a44f5885f32fa05844913 P 04088ee3bea440a7bd5ff772152b9410 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:18.025406 6323 ts_tablet_manager.cc:1398] T b284aa82594a44f5885f32fa05844913 P 04088ee3bea440a7bd5ff772152b9410: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:05:18.025717 6323 tablet_bootstrap.cc:492] T e7d5f02ccba34254a4d20951878b1214 P 04088ee3bea440a7bd5ff772152b9410: Bootstrap starting.
I20240501 14:05:18.026425 6324 tablet_bootstrap.cc:492] T b284aa82594a44f5885f32fa05844913 P e9c60449cf3c46f8aca4ab6c535302a4: No bootstrap required, opened a new log
I20240501 14:05:18.026743 6324 ts_tablet_manager.cc:1367] T b284aa82594a44f5885f32fa05844913 P e9c60449cf3c46f8aca4ab6c535302a4: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.004s
I20240501 14:05:18.027102 6323 tablet_bootstrap.cc:654] T e7d5f02ccba34254a4d20951878b1214 P 04088ee3bea440a7bd5ff772152b9410: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:18.028167 6324 raft_consensus.cc:348] T b284aa82594a44f5885f32fa05844913 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:18.028429 6324 raft_consensus.cc:374] T b284aa82594a44f5885f32fa05844913 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:18.028542 6324 raft_consensus.cc:729] T b284aa82594a44f5885f32fa05844913 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9c60449cf3c46f8aca4ab6c535302a4, State: Initialized, Role: FOLLOWER
I20240501 14:05:18.028836 6324 consensus_queue.cc:260] T b284aa82594a44f5885f32fa05844913 P e9c60449cf3c46f8aca4ab6c535302a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:18.029438 6324 ts_tablet_manager.cc:1398] T b284aa82594a44f5885f32fa05844913 P e9c60449cf3c46f8aca4ab6c535302a4: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240501 14:05:18.030385 6327 ts_tablet_manager.cc:1398] T e7d5f02ccba34254a4d20951878b1214 P 1c3fdc5272484ac09ef3e5e36a08f27c: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:05:18.030225 6326 tablet_bootstrap.cc:492] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf: No bootstrap required, opened a new log
I20240501 14:05:18.030709 6326 ts_tablet_manager.cc:1367] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.003s
I20240501 14:05:18.031412 6326 raft_consensus.cc:348] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } }
I20240501 14:05:18.032939 6327 tablet_bootstrap.cc:492] T ba6cba753e934e719c730593c0706346 P 1c3fdc5272484ac09ef3e5e36a08f27c: Bootstrap starting.
I20240501 14:05:18.033139 6326 raft_consensus.cc:374] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:18.033293 6326 raft_consensus.cc:729] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be56773105e744919601127a94e98adf, State: Initialized, Role: FOLLOWER
I20240501 14:05:18.033604 6326 consensus_queue.cc:260] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } }
I20240501 14:05:18.034337 6326 ts_tablet_manager.cc:1398] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240501 14:05:18.034891 6327 tablet_bootstrap.cc:654] T ba6cba753e934e719c730593c0706346 P 1c3fdc5272484ac09ef3e5e36a08f27c: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:18.036494 6327 tablet_bootstrap.cc:492] T ba6cba753e934e719c730593c0706346 P 1c3fdc5272484ac09ef3e5e36a08f27c: No bootstrap required, opened a new log
I20240501 14:05:18.036630 6327 ts_tablet_manager.cc:1367] T ba6cba753e934e719c730593c0706346 P 1c3fdc5272484ac09ef3e5e36a08f27c: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:05:18.036707 6323 tablet_bootstrap.cc:492] T e7d5f02ccba34254a4d20951878b1214 P 04088ee3bea440a7bd5ff772152b9410: No bootstrap required, opened a new log
I20240501 14:05:18.036847 6323 ts_tablet_manager.cc:1367] T e7d5f02ccba34254a4d20951878b1214 P 04088ee3bea440a7bd5ff772152b9410: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.003s
I20240501 14:05:18.037175 6327 raft_consensus.cc:348] T ba6cba753e934e719c730593c0706346 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } }
I20240501 14:05:18.037380 6327 raft_consensus.cc:374] T ba6cba753e934e719c730593c0706346 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:18.037453 6327 raft_consensus.cc:729] T ba6cba753e934e719c730593c0706346 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c3fdc5272484ac09ef3e5e36a08f27c, State: Initialized, Role: FOLLOWER
I20240501 14:05:18.037608 6323 raft_consensus.cc:348] T e7d5f02ccba34254a4d20951878b1214 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } }
I20240501 14:05:18.037675 6327 consensus_queue.cc:260] T ba6cba753e934e719c730593c0706346 P 1c3fdc5272484ac09ef3e5e36a08f27c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } }
I20240501 14:05:18.037798 6323 raft_consensus.cc:374] T e7d5f02ccba34254a4d20951878b1214 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:18.037886 6323 raft_consensus.cc:729] T e7d5f02ccba34254a4d20951878b1214 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04088ee3bea440a7bd5ff772152b9410, State: Initialized, Role: FOLLOWER
I20240501 14:05:18.038125 6327 ts_tablet_manager.cc:1398] T ba6cba753e934e719c730593c0706346 P 1c3fdc5272484ac09ef3e5e36a08f27c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20240501 14:05:18.038071 6323 consensus_queue.cc:260] T e7d5f02ccba34254a4d20951878b1214 P 04088ee3bea440a7bd5ff772152b9410 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } }
I20240501 14:05:18.038553 6323 ts_tablet_manager.cc:1398] T e7d5f02ccba34254a4d20951878b1214 P 04088ee3bea440a7bd5ff772152b9410: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:05:18.091825 6317 raft_consensus.cc:482] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:05:18.091987 6317 raft_consensus.cc:504] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } }
I20240501 14:05:18.092562 6317 leader_election.cc:288] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209), 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129:44033)
I20240501 14:05:18.092962 5254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7d5f02ccba34254a4d20951878b1214" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" is_pre_election: true
I20240501 14:05:18.093036 4992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7d5f02ccba34254a4d20951878b1214" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04088ee3bea440a7bd5ff772152b9410" is_pre_election: true
I20240501 14:05:18.093207 5254 raft_consensus.cc:2454] T e7d5f02ccba34254a4d20951878b1214 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 0.
I20240501 14:05:18.093233 4992 raft_consensus.cc:2454] T e7d5f02ccba34254a4d20951878b1214 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 0.
I20240501 14:05:18.093674 5319 leader_election.cc:302] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters:
I20240501 14:05:18.093922 6317 raft_consensus.cc:2781] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:05:18.094035 6317 raft_consensus.cc:482] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:05:18.094117 6317 raft_consensus.cc:3037] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:18.095373 6336 raft_consensus.cc:482] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:05:18.095537 6336 raft_consensus.cc:504] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:18.096213 6336 leader_election.cc:288] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129:44033), e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377)
I20240501 14:05:18.096534 4992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b284aa82594a44f5885f32fa05844913" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04088ee3bea440a7bd5ff772152b9410" is_pre_election: true
I20240501 14:05:18.096540 6317 raft_consensus.cc:504] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } }
I20240501 14:05:18.096722 4992 raft_consensus.cc:2454] T b284aa82594a44f5885f32fa05844913 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 0.
I20240501 14:05:18.096740 5126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b284aa82594a44f5885f32fa05844913" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" is_pre_election: true
I20240501 14:05:18.096961 5126 raft_consensus.cc:2454] T b284aa82594a44f5885f32fa05844913 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 0.
I20240501 14:05:18.097077 6317 leader_election.cc:288] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 1 election: Requested vote from peers 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209), 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129:44033)
I20240501 14:05:18.097358 5320 leader_election.cc:302] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04088ee3bea440a7bd5ff772152b9410, be56773105e744919601127a94e98adf; no voters:
I20240501 14:05:18.097608 6317 raft_consensus.cc:2781] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:05:18.097579 4992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7d5f02ccba34254a4d20951878b1214" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04088ee3bea440a7bd5ff772152b9410"
I20240501 14:05:18.097741 4992 raft_consensus.cc:3037] T e7d5f02ccba34254a4d20951878b1214 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:18.097739 6317 raft_consensus.cc:482] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:05:18.097743 5254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e7d5f02ccba34254a4d20951878b1214" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c"
I20240501 14:05:18.097859 6317 raft_consensus.cc:3037] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:18.097936 5254 raft_consensus.cc:3037] T e7d5f02ccba34254a4d20951878b1214 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:18.099813 4992 raft_consensus.cc:2454] T e7d5f02ccba34254a4d20951878b1214 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 1.
I20240501 14:05:18.100142 5320 leader_election.cc:302] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04088ee3bea440a7bd5ff772152b9410, be56773105e744919601127a94e98adf; no voters:
I20240501 14:05:18.100379 6336 raft_consensus.cc:2781] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:05:18.100561 5254 raft_consensus.cc:2454] T e7d5f02ccba34254a4d20951878b1214 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 1.
I20240501 14:05:18.101233 6317 raft_consensus.cc:504] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:18.101511 6336 raft_consensus.cc:686] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [term 1 LEADER]: Becoming Leader. State: Replica: be56773105e744919601127a94e98adf, State: Running, Role: LEADER
I20240501 14:05:18.101920 6317 leader_election.cc:288] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 1 election: Requested vote from peers 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129:44033), e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377)
I20240501 14:05:18.102321 4992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b284aa82594a44f5885f32fa05844913" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04088ee3bea440a7bd5ff772152b9410"
I20240501 14:05:18.102265 5126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b284aa82594a44f5885f32fa05844913" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9c60449cf3c46f8aca4ab6c535302a4"
I20240501 14:05:18.102483 5126 raft_consensus.cc:3037] T b284aa82594a44f5885f32fa05844913 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:18.102506 4992 raft_consensus.cc:3037] T b284aa82594a44f5885f32fa05844913 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:18.104440 4992 raft_consensus.cc:2454] T b284aa82594a44f5885f32fa05844913 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 1.
I20240501 14:05:18.105131 5126 raft_consensus.cc:2454] T b284aa82594a44f5885f32fa05844913 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 1.
I20240501 14:05:18.102025 6336 consensus_queue.cc:237] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } }
I20240501 14:05:18.104774 5320 leader_election.cc:302] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04088ee3bea440a7bd5ff772152b9410, be56773105e744919601127a94e98adf; no voters:
I20240501 14:05:18.105710 6317 raft_consensus.cc:2781] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:05:18.105865 6317 raft_consensus.cc:686] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf [term 1 LEADER]: Becoming Leader. State: Replica: be56773105e744919601127a94e98adf, State: Running, Role: LEADER
I20240501 14:05:18.106120 6317 consensus_queue.cc:237] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:18.109165 4851 catalog_manager.cc:5484] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf reported cstate change: term changed from 0 to 1, leader changed from <none> to be56773105e744919601127a94e98adf (127.29.124.132). New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } health_report { overall_health: UNKNOWN } } }
I20240501 14:05:18.113741 4851 catalog_manager.cc:5484] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf reported cstate change: term changed from 0 to 1, leader changed from <none> to be56773105e744919601127a94e98adf (127.29.124.132). New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } health_report { overall_health: HEALTHY } } }
I20240501 14:05:18.269433 6222 raft_consensus.cc:482] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:05:18.269615 6222 raft_consensus.cc:504] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } }
I20240501 14:05:18.270326 6222 leader_election.cc:288] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209), e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377)
I20240501 14:05:18.270967 5126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba6cba753e934e719c730593c0706346" candidate_uuid: "04088ee3bea440a7bd5ff772152b9410" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" is_pre_election: true
I20240501 14:05:18.271216 5126 raft_consensus.cc:2454] T ba6cba753e934e719c730593c0706346 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04088ee3bea440a7bd5ff772152b9410 in term 0.
I20240501 14:05:18.271661 4929 leader_election.cc:302] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04088ee3bea440a7bd5ff772152b9410, e9c60449cf3c46f8aca4ab6c535302a4; no voters:
I20240501 14:05:18.271978 6222 raft_consensus.cc:2781] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:05:18.272114 6222 raft_consensus.cc:482] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:05:18.272241 6222 raft_consensus.cc:3037] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:18.274585 6222 raft_consensus.cc:504] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } }
I20240501 14:05:18.275300 6222 leader_election.cc:288] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 [CANDIDATE]: Term 1 election: Requested vote from peers 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209), e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377)
I20240501 14:05:18.275803 5126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba6cba753e934e719c730593c0706346" candidate_uuid: "04088ee3bea440a7bd5ff772152b9410" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9c60449cf3c46f8aca4ab6c535302a4"
I20240501 14:05:18.276026 5126 raft_consensus.cc:3037] T ba6cba753e934e719c730593c0706346 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:18.277113 5254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba6cba753e934e719c730593c0706346" candidate_uuid: "04088ee3bea440a7bd5ff772152b9410" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" is_pre_election: true
I20240501 14:05:18.277135 5256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba6cba753e934e719c730593c0706346" candidate_uuid: "04088ee3bea440a7bd5ff772152b9410" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c"
I20240501 14:05:18.277410 5254 raft_consensus.cc:2454] T ba6cba753e934e719c730593c0706346 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04088ee3bea440a7bd5ff772152b9410 in term 0.
I20240501 14:05:18.278116 5126 raft_consensus.cc:2454] T ba6cba753e934e719c730593c0706346 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04088ee3bea440a7bd5ff772152b9410 in term 1.
I20240501 14:05:18.278445 4929 leader_election.cc:302] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 04088ee3bea440a7bd5ff772152b9410, e9c60449cf3c46f8aca4ab6c535302a4; no voters: 1c3fdc5272484ac09ef3e5e36a08f27c
I20240501 14:05:18.278709 6222 raft_consensus.cc:2781] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:05:18.279019 6222 raft_consensus.cc:686] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEADER]: Becoming Leader. State: Replica: 04088ee3bea440a7bd5ff772152b9410, State: Running, Role: LEADER
I20240501 14:05:18.279309 6222 consensus_queue.cc:237] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } }
I20240501 14:05:18.282121 4851 catalog_manager.cc:5484] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 reported cstate change: term changed from 0 to 1, leader changed from <none> to 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129). New cstate: current_term: 1 leader_uuid: "04088ee3bea440a7bd5ff772152b9410" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: UNKNOWN } } }
I20240501 14:05:18.514150 6317 consensus_queue.cc:1035] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:05:18.519932 6317 consensus_queue.cc:1035] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:05:18.604593 6317 consensus_queue.cc:1035] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:05:18.610024 6317 consensus_queue.cc:1035] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:05:18.689918 6344 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:05:18.690258 6344 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:05:18.697687 6344 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:05:18.707495 6222 consensus_queue.cc:1035] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:05:18.711649 5254 raft_consensus.cc:3037] T ba6cba753e934e719c730593c0706346 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:18.715317 6362 consensus_queue.cc:1035] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_4'
INSERT report
rows total: 2000
time total: 212.075 ms
time per row: 0.106038 ms
I20240501 14:05:19.051128 4845 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55620:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:05:19.060923 5106 tablet_service.cc:1467] Processing CreateTablet for tablet f9cb784f60774ea7b6e70ccfebf867a8 (DEFAULT_TABLE table=rebalance_test_table_5 [id=3794b1ccb5004f7cb92a229ef6ce3595]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:19.060956 5236 tablet_service.cc:1467] Processing CreateTablet for tablet f9cb784f60774ea7b6e70ccfebf867a8 (DEFAULT_TABLE table=rebalance_test_table_5 [id=3794b1ccb5004f7cb92a229ef6ce3595]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:19.061347 5236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9cb784f60774ea7b6e70ccfebf867a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:19.061378 5106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9cb784f60774ea7b6e70ccfebf867a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:19.066103 5366 tablet_service.cc:1467] Processing CreateTablet for tablet f9cb784f60774ea7b6e70ccfebf867a8 (DEFAULT_TABLE table=rebalance_test_table_5 [id=3794b1ccb5004f7cb92a229ef6ce3595]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:19.066336 6407 tablet_bootstrap.cc:492] T f9cb784f60774ea7b6e70ccfebf867a8 P 1c3fdc5272484ac09ef3e5e36a08f27c: Bootstrap starting.
I20240501 14:05:19.066553 5366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9cb784f60774ea7b6e70ccfebf867a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:19.066968 5234 tablet_service.cc:1467] Processing CreateTablet for tablet 2c39b53867e047faaf24cc6f87a8100c (DEFAULT_TABLE table=rebalance_test_table_5 [id=3794b1ccb5004f7cb92a229ef6ce3595]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:19.067365 5234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c39b53867e047faaf24cc6f87a8100c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:19.068089 6407 tablet_bootstrap.cc:654] T f9cb784f60774ea7b6e70ccfebf867a8 P 1c3fdc5272484ac09ef3e5e36a08f27c: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:19.070006 5105 tablet_service.cc:1467] Processing CreateTablet for tablet 2c39b53867e047faaf24cc6f87a8100c (DEFAULT_TABLE table=rebalance_test_table_5 [id=3794b1ccb5004f7cb92a229ef6ce3595]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:19.070423 5105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c39b53867e047faaf24cc6f87a8100c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:19.075260 5365 tablet_service.cc:1467] Processing CreateTablet for tablet 2c39b53867e047faaf24cc6f87a8100c (DEFAULT_TABLE table=rebalance_test_table_5 [id=3794b1ccb5004f7cb92a229ef6ce3595]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:19.075762 5365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c39b53867e047faaf24cc6f87a8100c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:19.078012 4974 tablet_service.cc:1467] Processing CreateTablet for tablet 9fd94635797d447cbdc9181ee91f465b (DEFAULT_TABLE table=rebalance_test_table_5 [id=3794b1ccb5004f7cb92a229ef6ce3595]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:19.078474 4974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fd94635797d447cbdc9181ee91f465b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:19.081728 6410 tablet_bootstrap.cc:492] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4: Bootstrap starting.
I20240501 14:05:19.083596 6410 tablet_bootstrap.cc:654] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:19.093988 5236 tablet_service.cc:1467] Processing CreateTablet for tablet 9fd94635797d447cbdc9181ee91f465b (DEFAULT_TABLE table=rebalance_test_table_5 [id=3794b1ccb5004f7cb92a229ef6ce3595]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:19.094449 5236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fd94635797d447cbdc9181ee91f465b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:19.095716 6412 tablet_bootstrap.cc:492] T 9fd94635797d447cbdc9181ee91f465b P 04088ee3bea440a7bd5ff772152b9410: Bootstrap starting.
I20240501 14:05:19.097684 6412 tablet_bootstrap.cc:654] T 9fd94635797d447cbdc9181ee91f465b P 04088ee3bea440a7bd5ff772152b9410: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:19.100350 6410 tablet_bootstrap.cc:492] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4: No bootstrap required, opened a new log
I20240501 14:05:19.100941 6412 tablet_bootstrap.cc:492] T 9fd94635797d447cbdc9181ee91f465b P 04088ee3bea440a7bd5ff772152b9410: No bootstrap required, opened a new log
I20240501 14:05:19.101096 6412 ts_tablet_manager.cc:1367] T 9fd94635797d447cbdc9181ee91f465b P 04088ee3bea440a7bd5ff772152b9410: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 14:05:19.093845 5106 tablet_service.cc:1467] Processing CreateTablet for tablet 9fd94635797d447cbdc9181ee91f465b (DEFAULT_TABLE table=rebalance_test_table_5 [id=3794b1ccb5004f7cb92a229ef6ce3595]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:19.102674 5106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fd94635797d447cbdc9181ee91f465b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:19.102854 6407 tablet_bootstrap.cc:492] T f9cb784f60774ea7b6e70ccfebf867a8 P 1c3fdc5272484ac09ef3e5e36a08f27c: No bootstrap required, opened a new log
I20240501 14:05:19.103005 6407 ts_tablet_manager.cc:1367] T f9cb784f60774ea7b6e70ccfebf867a8 P 1c3fdc5272484ac09ef3e5e36a08f27c: Time spent bootstrapping tablet: real 0.037s user 0.024s sys 0.012s
I20240501 14:05:19.103925 6407 raft_consensus.cc:348] T f9cb784f60774ea7b6e70ccfebf867a8 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:19.104190 6407 raft_consensus.cc:374] T f9cb784f60774ea7b6e70ccfebf867a8 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:19.104324 6407 raft_consensus.cc:729] T f9cb784f60774ea7b6e70ccfebf867a8 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c3fdc5272484ac09ef3e5e36a08f27c, State: Initialized, Role: FOLLOWER
I20240501 14:05:19.104635 6407 consensus_queue.cc:260] T f9cb784f60774ea7b6e70ccfebf867a8 P 1c3fdc5272484ac09ef3e5e36a08f27c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:19.105391 6407 ts_tablet_manager.cc:1398] T f9cb784f60774ea7b6e70ccfebf867a8 P 1c3fdc5272484ac09ef3e5e36a08f27c: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240501 14:05:19.105680 6407 tablet_bootstrap.cc:492] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c: Bootstrap starting.
I20240501 14:05:19.107338 6407 tablet_bootstrap.cc:654] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:19.108067 6412 raft_consensus.cc:348] T 9fd94635797d447cbdc9181ee91f465b P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:19.108278 6412 raft_consensus.cc:374] T 9fd94635797d447cbdc9181ee91f465b P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:19.108390 6412 raft_consensus.cc:729] T 9fd94635797d447cbdc9181ee91f465b P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04088ee3bea440a7bd5ff772152b9410, State: Initialized, Role: FOLLOWER
I20240501 14:05:19.109337 6409 tablet_bootstrap.cc:492] T f9cb784f60774ea7b6e70ccfebf867a8 P be56773105e744919601127a94e98adf: Bootstrap starting.
I20240501 14:05:19.110796 6412 consensus_queue.cc:260] T 9fd94635797d447cbdc9181ee91f465b P 04088ee3bea440a7bd5ff772152b9410 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:19.111119 6409 tablet_bootstrap.cc:654] T f9cb784f60774ea7b6e70ccfebf867a8 P be56773105e744919601127a94e98adf: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:19.112905 6412 ts_tablet_manager.cc:1398] T 9fd94635797d447cbdc9181ee91f465b P 04088ee3bea440a7bd5ff772152b9410: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240501 14:05:19.115082 6409 tablet_bootstrap.cc:492] T f9cb784f60774ea7b6e70ccfebf867a8 P be56773105e744919601127a94e98adf: No bootstrap required, opened a new log
I20240501 14:05:19.115288 6409 ts_tablet_manager.cc:1367] T f9cb784f60774ea7b6e70ccfebf867a8 P be56773105e744919601127a94e98adf: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240501 14:05:19.115324 6407 tablet_bootstrap.cc:492] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c: No bootstrap required, opened a new log
I20240501 14:05:19.115471 6407 ts_tablet_manager.cc:1367] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.007s
I20240501 14:05:19.116266 6407 raft_consensus.cc:348] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:19.116293 6409 raft_consensus.cc:348] T f9cb784f60774ea7b6e70ccfebf867a8 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:19.116477 6409 raft_consensus.cc:374] T f9cb784f60774ea7b6e70ccfebf867a8 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:19.116494 6407 raft_consensus.cc:374] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:19.116564 6407 raft_consensus.cc:729] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c3fdc5272484ac09ef3e5e36a08f27c, State: Initialized, Role: FOLLOWER
I20240501 14:05:19.116612 6409 raft_consensus.cc:729] T f9cb784f60774ea7b6e70ccfebf867a8 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be56773105e744919601127a94e98adf, State: Initialized, Role: FOLLOWER
I20240501 14:05:19.116818 6407 consensus_queue.cc:260] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:19.116866 6409 consensus_queue.cc:260] T f9cb784f60774ea7b6e70ccfebf867a8 P be56773105e744919601127a94e98adf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:19.117435 6409 ts_tablet_manager.cc:1398] T f9cb784f60774ea7b6e70ccfebf867a8 P be56773105e744919601127a94e98adf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:05:19.118132 6407 ts_tablet_manager.cc:1398] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:05:19.117758 6409 tablet_bootstrap.cc:492] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf: Bootstrap starting.
I20240501 14:05:19.118430 6407 tablet_bootstrap.cc:492] T 2c39b53867e047faaf24cc6f87a8100c P 1c3fdc5272484ac09ef3e5e36a08f27c: Bootstrap starting.
I20240501 14:05:19.119722 6409 tablet_bootstrap.cc:654] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:19.120662 6407 tablet_bootstrap.cc:654] T 2c39b53867e047faaf24cc6f87a8100c P 1c3fdc5272484ac09ef3e5e36a08f27c: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:19.121573 6410 ts_tablet_manager.cc:1367] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4: Time spent bootstrapping tablet: real 0.040s user 0.002s sys 0.008s
I20240501 14:05:19.122431 6410 raft_consensus.cc:348] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:19.122646 6410 raft_consensus.cc:374] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:19.122741 6410 raft_consensus.cc:729] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9c60449cf3c46f8aca4ab6c535302a4, State: Initialized, Role: FOLLOWER
I20240501 14:05:19.122980 6410 consensus_queue.cc:260] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:19.123517 6410 ts_tablet_manager.cc:1398] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:05:19.123795 6410 tablet_bootstrap.cc:492] T 2c39b53867e047faaf24cc6f87a8100c P e9c60449cf3c46f8aca4ab6c535302a4: Bootstrap starting.
I20240501 14:05:19.126888 6410 tablet_bootstrap.cc:654] T 2c39b53867e047faaf24cc6f87a8100c P e9c60449cf3c46f8aca4ab6c535302a4: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:19.128103 6407 tablet_bootstrap.cc:492] T 2c39b53867e047faaf24cc6f87a8100c P 1c3fdc5272484ac09ef3e5e36a08f27c: No bootstrap required, opened a new log
I20240501 14:05:19.128263 6407 ts_tablet_manager.cc:1367] T 2c39b53867e047faaf24cc6f87a8100c P 1c3fdc5272484ac09ef3e5e36a08f27c: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.005s
I20240501 14:05:19.129076 6407 raft_consensus.cc:348] T 2c39b53867e047faaf24cc6f87a8100c P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:19.129312 6407 raft_consensus.cc:374] T 2c39b53867e047faaf24cc6f87a8100c P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:19.129417 6407 raft_consensus.cc:729] T 2c39b53867e047faaf24cc6f87a8100c P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c3fdc5272484ac09ef3e5e36a08f27c, State: Initialized, Role: FOLLOWER
I20240501 14:05:19.129675 6407 consensus_queue.cc:260] T 2c39b53867e047faaf24cc6f87a8100c P 1c3fdc5272484ac09ef3e5e36a08f27c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:19.130347 6407 ts_tablet_manager.cc:1398] T 2c39b53867e047faaf24cc6f87a8100c P 1c3fdc5272484ac09ef3e5e36a08f27c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:05:19.131503 6409 tablet_bootstrap.cc:492] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf: No bootstrap required, opened a new log
I20240501 14:05:19.131711 6410 tablet_bootstrap.cc:492] T 2c39b53867e047faaf24cc6f87a8100c P e9c60449cf3c46f8aca4ab6c535302a4: No bootstrap required, opened a new log
I20240501 14:05:19.131713 6409 ts_tablet_manager.cc:1367] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.009s
I20240501 14:05:19.131899 6410 ts_tablet_manager.cc:1367] T 2c39b53867e047faaf24cc6f87a8100c P e9c60449cf3c46f8aca4ab6c535302a4: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.005s
I20240501 14:05:19.132464 6409 raft_consensus.cc:348] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:19.132632 6409 raft_consensus.cc:374] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:19.132575 6410 raft_consensus.cc:348] T 2c39b53867e047faaf24cc6f87a8100c P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:19.132709 6409 raft_consensus.cc:729] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be56773105e744919601127a94e98adf, State: Initialized, Role: FOLLOWER
I20240501 14:05:19.132819 6410 raft_consensus.cc:374] T 2c39b53867e047faaf24cc6f87a8100c P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:19.132910 6410 raft_consensus.cc:729] T 2c39b53867e047faaf24cc6f87a8100c P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9c60449cf3c46f8aca4ab6c535302a4, State: Initialized, Role: FOLLOWER
I20240501 14:05:19.132972 6409 consensus_queue.cc:260] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:19.133152 6410 consensus_queue.cc:260] T 2c39b53867e047faaf24cc6f87a8100c P e9c60449cf3c46f8aca4ab6c535302a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:19.133700 6410 ts_tablet_manager.cc:1398] T 2c39b53867e047faaf24cc6f87a8100c P e9c60449cf3c46f8aca4ab6c535302a4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:05:19.133805 6409 ts_tablet_manager.cc:1398] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:05:19.134178 6410 tablet_bootstrap.cc:492] T 9fd94635797d447cbdc9181ee91f465b P e9c60449cf3c46f8aca4ab6c535302a4: Bootstrap starting.
I20240501 14:05:19.135649 6410 tablet_bootstrap.cc:654] T 9fd94635797d447cbdc9181ee91f465b P e9c60449cf3c46f8aca4ab6c535302a4: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:19.138378 6410 tablet_bootstrap.cc:492] T 9fd94635797d447cbdc9181ee91f465b P e9c60449cf3c46f8aca4ab6c535302a4: No bootstrap required, opened a new log
I20240501 14:05:19.138491 6410 ts_tablet_manager.cc:1367] T 9fd94635797d447cbdc9181ee91f465b P e9c60449cf3c46f8aca4ab6c535302a4: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:05:19.139017 6410 raft_consensus.cc:348] T 9fd94635797d447cbdc9181ee91f465b P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:19.139159 6410 raft_consensus.cc:374] T 9fd94635797d447cbdc9181ee91f465b P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:19.139241 6410 raft_consensus.cc:729] T 9fd94635797d447cbdc9181ee91f465b P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9c60449cf3c46f8aca4ab6c535302a4, State: Initialized, Role: FOLLOWER
I20240501 14:05:19.139545 6410 consensus_queue.cc:260] T 9fd94635797d447cbdc9181ee91f465b P e9c60449cf3c46f8aca4ab6c535302a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:19.140060 6410 ts_tablet_manager.cc:1398] T 9fd94635797d447cbdc9181ee91f465b P e9c60449cf3c46f8aca4ab6c535302a4: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:05:19.182076 6296 raft_consensus.cc:482] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:05:19.182235 6296 raft_consensus.cc:504] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:19.182951 6296 leader_election.cc:288] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209), be56773105e744919601127a94e98adf (127.29.124.132:35459)
I20240501 14:05:19.183387 5256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9cb784f60774ea7b6e70ccfebf867a8" candidate_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" is_pre_election: true
I20240501 14:05:19.183466 5385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9cb784f60774ea7b6e70ccfebf867a8" candidate_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf" is_pre_election: true
I20240501 14:05:19.183605 5256 raft_consensus.cc:2454] T f9cb784f60774ea7b6e70ccfebf867a8 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9c60449cf3c46f8aca4ab6c535302a4 in term 0.
I20240501 14:05:19.183701 5385 raft_consensus.cc:2454] T f9cb784f60774ea7b6e70ccfebf867a8 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9c60449cf3c46f8aca4ab6c535302a4 in term 0.
I20240501 14:05:19.184084 5059 leader_election.cc:302] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, e9c60449cf3c46f8aca4ab6c535302a4; no voters:
I20240501 14:05:19.184386 6296 raft_consensus.cc:2781] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:05:19.184509 6296 raft_consensus.cc:482] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:05:19.184603 6296 raft_consensus.cc:3037] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:19.186496 6296 raft_consensus.cc:504] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:19.186962 6296 leader_election.cc:288] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 [CANDIDATE]: Term 1 election: Requested vote from peers 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209), be56773105e744919601127a94e98adf (127.29.124.132:35459)
I20240501 14:05:19.187366 5256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9cb784f60774ea7b6e70ccfebf867a8" candidate_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c"
I20240501 14:05:19.187404 5385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9cb784f60774ea7b6e70ccfebf867a8" candidate_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf"
I20240501 14:05:19.187546 5256 raft_consensus.cc:3037] T f9cb784f60774ea7b6e70ccfebf867a8 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:19.187568 5385 raft_consensus.cc:3037] T f9cb784f60774ea7b6e70ccfebf867a8 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:19.189321 5256 raft_consensus.cc:2454] T f9cb784f60774ea7b6e70ccfebf867a8 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9c60449cf3c46f8aca4ab6c535302a4 in term 1.
I20240501 14:05:19.189348 5385 raft_consensus.cc:2454] T f9cb784f60774ea7b6e70ccfebf867a8 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9c60449cf3c46f8aca4ab6c535302a4 in term 1.
I20240501 14:05:19.189669 5059 leader_election.cc:302] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, e9c60449cf3c46f8aca4ab6c535302a4; no voters:
I20240501 14:05:19.189944 6296 raft_consensus.cc:2781] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:05:19.190323 6296 raft_consensus.cc:686] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 LEADER]: Becoming Leader. State: Replica: e9c60449cf3c46f8aca4ab6c535302a4, State: Running, Role: LEADER
I20240501 14:05:19.190568 6296 consensus_queue.cc:237] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:19.193136 4851 catalog_manager.cc:5484] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130). New cstate: current_term: 1 leader_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } health_report { overall_health: UNKNOWN } } }
I20240501 14:05:19.202836 6381 raft_consensus.cc:482] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:05:19.202987 6381 raft_consensus.cc:504] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:19.203593 6381 leader_election.cc:288] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129:44033), e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377)
I20240501 14:05:19.204146 5125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fd94635797d447cbdc9181ee91f465b" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" is_pre_election: true
I20240501 14:05:19.204182 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fd94635797d447cbdc9181ee91f465b" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04088ee3bea440a7bd5ff772152b9410" is_pre_election: true
I20240501 14:05:19.204388 4995 raft_consensus.cc:2454] T 9fd94635797d447cbdc9181ee91f465b P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 0.
I20240501 14:05:19.204385 5125 raft_consensus.cc:2454] T 9fd94635797d447cbdc9181ee91f465b P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 0.
I20240501 14:05:19.204779 5190 leader_election.cc:302] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04088ee3bea440a7bd5ff772152b9410, 1c3fdc5272484ac09ef3e5e36a08f27c; no voters:
I20240501 14:05:19.205017 6381 raft_consensus.cc:2781] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:05:19.205132 6381 raft_consensus.cc:482] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:05:19.205216 6381 raft_consensus.cc:3037] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:19.206974 6381 raft_consensus.cc:504] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:19.207464 6381 leader_election.cc:288] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 1 election: Requested vote from peers 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129:44033), e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377)
I20240501 14:05:19.207867 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fd94635797d447cbdc9181ee91f465b" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04088ee3bea440a7bd5ff772152b9410"
I20240501 14:05:19.207917 5125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fd94635797d447cbdc9181ee91f465b" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9c60449cf3c46f8aca4ab6c535302a4"
I20240501 14:05:19.208086 5125 raft_consensus.cc:3037] T 9fd94635797d447cbdc9181ee91f465b P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:19.208082 4995 raft_consensus.cc:3037] T 9fd94635797d447cbdc9181ee91f465b P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:19.210748 5125 raft_consensus.cc:2454] T 9fd94635797d447cbdc9181ee91f465b P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 1.
I20240501 14:05:19.210914 4995 raft_consensus.cc:2454] T 9fd94635797d447cbdc9181ee91f465b P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 1.
I20240501 14:05:19.211189 5190 leader_election.cc:302] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, e9c60449cf3c46f8aca4ab6c535302a4; no voters:
I20240501 14:05:19.211454 6381 raft_consensus.cc:2781] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:05:19.211609 6381 raft_consensus.cc:686] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEADER]: Becoming Leader. State: Replica: 1c3fdc5272484ac09ef3e5e36a08f27c, State: Running, Role: LEADER
I20240501 14:05:19.211874 6381 consensus_queue.cc:237] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:19.214947 4845 catalog_manager.cc:5484] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131). New cstate: current_term: 1 leader_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } health_report { overall_health: HEALTHY } } }
I20240501 14:05:19.311620 6400 raft_consensus.cc:482] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:05:19.311776 6400 raft_consensus.cc:504] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:19.312402 6400 leader_election.cc:288] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209), e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377)
I20240501 14:05:19.312860 5256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c39b53867e047faaf24cc6f87a8100c" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" is_pre_election: true
I20240501 14:05:19.312902 5125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c39b53867e047faaf24cc6f87a8100c" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" is_pre_election: true
I20240501 14:05:19.313066 5256 raft_consensus.cc:2454] T 2c39b53867e047faaf24cc6f87a8100c P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 0.
I20240501 14:05:19.313086 5125 raft_consensus.cc:2454] T 2c39b53867e047faaf24cc6f87a8100c P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 0.
I20240501 14:05:19.313464 5319 leader_election.cc:302] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters:
I20240501 14:05:19.313724 6400 raft_consensus.cc:2781] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:05:19.313818 6400 raft_consensus.cc:482] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:05:19.313881 6400 raft_consensus.cc:3037] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:19.315725 6400 raft_consensus.cc:504] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:19.316222 6400 leader_election.cc:288] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf [CANDIDATE]: Term 1 election: Requested vote from peers 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209), e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377)
I20240501 14:05:19.316560 5256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c39b53867e047faaf24cc6f87a8100c" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c"
I20240501 14:05:19.316718 5256 raft_consensus.cc:3037] T 2c39b53867e047faaf24cc6f87a8100c P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:19.316661 5125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c39b53867e047faaf24cc6f87a8100c" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9c60449cf3c46f8aca4ab6c535302a4"
I20240501 14:05:19.316826 5125 raft_consensus.cc:3037] T 2c39b53867e047faaf24cc6f87a8100c P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:19.318598 5256 raft_consensus.cc:2454] T 2c39b53867e047faaf24cc6f87a8100c P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 1.
I20240501 14:05:19.318684 5125 raft_consensus.cc:2454] T 2c39b53867e047faaf24cc6f87a8100c P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 1.
I20240501 14:05:19.319031 5319 leader_election.cc:302] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters:
I20240501 14:05:19.319310 6400 raft_consensus.cc:2781] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:05:19.319430 6400 raft_consensus.cc:686] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf [term 1 LEADER]: Becoming Leader. State: Replica: be56773105e744919601127a94e98adf, State: Running, Role: LEADER
I20240501 14:05:19.319691 6400 consensus_queue.cc:237] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:19.322602 4845 catalog_manager.cc:5484] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf reported cstate change: term changed from 0 to 1, leader changed from <none> to be56773105e744919601127a94e98adf (127.29.124.132). New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } health_report { overall_health: HEALTHY } } }
W20240501 14:05:19.686308 6424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:05:19.686630 6424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:05:19.693989 6424 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
I20240501 14:05:19.740129 6381 consensus_queue.cc:1035] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:05:19.740244 6296 consensus_queue.cc:1035] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:05:19.761638 6448 consensus_queue.cc:1035] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:05:19.767001 6296 consensus_queue.cc:1035] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:05:19.819167 6400 consensus_queue.cc:1035] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:05:19.843313 6467 consensus_queue.cc:1035] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:05:19.954939 5319 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.124.131:33209, user_credentials={real_user=slave}} blocked reactor thread for 52629.5us
INSERT report
rows total: 2000
time total: 271.213 ms
time per row: 0.135607 ms
I20240501 14:05:20.102927 4849 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55620:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:05:20.112761 5106 tablet_service.cc:1467] Processing CreateTablet for tablet ac4b234f794149e4b6def7a3b358942d (DEFAULT_TABLE table=rebalance_test_table_6 [id=3c1513e944ee4b518373c4efa5e200cc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:20.113224 5106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac4b234f794149e4b6def7a3b358942d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:20.113523 5234 tablet_service.cc:1467] Processing CreateTablet for tablet de6c7753c8564107b858876fcfa5924d (DEFAULT_TABLE table=rebalance_test_table_6 [id=3c1513e944ee4b518373c4efa5e200cc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:20.113925 5234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de6c7753c8564107b858876fcfa5924d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:20.117976 5366 tablet_service.cc:1467] Processing CreateTablet for tablet ac4b234f794149e4b6def7a3b358942d (DEFAULT_TABLE table=rebalance_test_table_6 [id=3c1513e944ee4b518373c4efa5e200cc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:20.118242 4974 tablet_service.cc:1467] Processing CreateTablet for tablet ac4b234f794149e4b6def7a3b358942d (DEFAULT_TABLE table=rebalance_test_table_6 [id=3c1513e944ee4b518373c4efa5e200cc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:20.118376 5366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac4b234f794149e4b6def7a3b358942d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:20.118666 4974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac4b234f794149e4b6def7a3b358942d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:20.119640 4975 tablet_service.cc:1467] Processing CreateTablet for tablet 8cc5eec4600640f7a293877c825f8d96 (DEFAULT_TABLE table=rebalance_test_table_6 [id=3c1513e944ee4b518373c4efa5e200cc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:20.119998 4975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cc5eec4600640f7a293877c825f8d96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:20.121634 4973 tablet_service.cc:1467] Processing CreateTablet for tablet de6c7753c8564107b858876fcfa5924d (DEFAULT_TABLE table=rebalance_test_table_6 [id=3c1513e944ee4b518373c4efa5e200cc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:20.121986 4973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de6c7753c8564107b858876fcfa5924d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:20.126125 5365 tablet_service.cc:1467] Processing CreateTablet for tablet de6c7753c8564107b858876fcfa5924d (DEFAULT_TABLE table=rebalance_test_table_6 [id=3c1513e944ee4b518373c4efa5e200cc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:20.126550 5365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de6c7753c8564107b858876fcfa5924d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:20.129604 5364 tablet_service.cc:1467] Processing CreateTablet for tablet 8cc5eec4600640f7a293877c825f8d96 (DEFAULT_TABLE table=rebalance_test_table_6 [id=3c1513e944ee4b518373c4efa5e200cc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:20.129941 5364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cc5eec4600640f7a293877c825f8d96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:20.133353 6498 tablet_bootstrap.cc:492] T ac4b234f794149e4b6def7a3b358942d P 04088ee3bea440a7bd5ff772152b9410: Bootstrap starting.
I20240501 14:05:20.134001 5105 tablet_service.cc:1467] Processing CreateTablet for tablet 8cc5eec4600640f7a293877c825f8d96 (DEFAULT_TABLE table=rebalance_test_table_6 [id=3c1513e944ee4b518373c4efa5e200cc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:20.134402 5105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cc5eec4600640f7a293877c825f8d96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:20.135165 6498 tablet_bootstrap.cc:654] T ac4b234f794149e4b6def7a3b358942d P 04088ee3bea440a7bd5ff772152b9410: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:20.140623 6497 tablet_bootstrap.cc:492] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c: Bootstrap starting.
I20240501 14:05:20.142634 6497 tablet_bootstrap.cc:654] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:20.147622 6498 tablet_bootstrap.cc:492] T ac4b234f794149e4b6def7a3b358942d P 04088ee3bea440a7bd5ff772152b9410: No bootstrap required, opened a new log
I20240501 14:05:20.147851 6498 ts_tablet_manager.cc:1367] T ac4b234f794149e4b6def7a3b358942d P 04088ee3bea440a7bd5ff772152b9410: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.007s
I20240501 14:05:20.148761 6498 raft_consensus.cc:348] T ac4b234f794149e4b6def7a3b358942d P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } }
I20240501 14:05:20.149091 6498 raft_consensus.cc:374] T ac4b234f794149e4b6def7a3b358942d P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:20.149328 6498 raft_consensus.cc:729] T ac4b234f794149e4b6def7a3b358942d P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04088ee3bea440a7bd5ff772152b9410, State: Initialized, Role: FOLLOWER
I20240501 14:05:20.149693 6498 consensus_queue.cc:260] T ac4b234f794149e4b6def7a3b358942d P 04088ee3bea440a7bd5ff772152b9410 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } }
I20240501 14:05:20.151162 6498 ts_tablet_manager.cc:1398] T ac4b234f794149e4b6def7a3b358942d P 04088ee3bea440a7bd5ff772152b9410: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:05:20.151476 6498 tablet_bootstrap.cc:492] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410: Bootstrap starting.
I20240501 14:05:20.153105 6498 tablet_bootstrap.cc:654] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:20.160665 6502 tablet_bootstrap.cc:492] T de6c7753c8564107b858876fcfa5924d P be56773105e744919601127a94e98adf: Bootstrap starting.
I20240501 14:05:20.162350 6502 tablet_bootstrap.cc:654] T de6c7753c8564107b858876fcfa5924d P be56773105e744919601127a94e98adf: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:20.162422 6498 tablet_bootstrap.cc:492] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410: No bootstrap required, opened a new log
I20240501 14:05:20.162565 6498 ts_tablet_manager.cc:1367] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.007s
I20240501 14:05:20.163414 6498 raft_consensus.cc:348] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } }
I20240501 14:05:20.163647 6498 raft_consensus.cc:374] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:20.163715 6498 raft_consensus.cc:729] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04088ee3bea440a7bd5ff772152b9410, State: Initialized, Role: FOLLOWER
I20240501 14:05:20.163877 6501 tablet_bootstrap.cc:492] T 8cc5eec4600640f7a293877c825f8d96 P e9c60449cf3c46f8aca4ab6c535302a4: Bootstrap starting.
I20240501 14:05:20.163976 6498 consensus_queue.cc:260] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } }
I20240501 14:05:20.164633 6498 ts_tablet_manager.cc:1398] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:05:20.164924 6498 tablet_bootstrap.cc:492] T de6c7753c8564107b858876fcfa5924d P 04088ee3bea440a7bd5ff772152b9410: Bootstrap starting.
I20240501 14:05:20.165639 6501 tablet_bootstrap.cc:654] T 8cc5eec4600640f7a293877c825f8d96 P e9c60449cf3c46f8aca4ab6c535302a4: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:20.166374 6498 tablet_bootstrap.cc:654] T de6c7753c8564107b858876fcfa5924d P 04088ee3bea440a7bd5ff772152b9410: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:20.167771 6497 tablet_bootstrap.cc:492] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c: No bootstrap required, opened a new log
I20240501 14:05:20.167922 6497 ts_tablet_manager.cc:1367] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c: Time spent bootstrapping tablet: real 0.027s user 0.009s sys 0.005s
I20240501 14:05:20.168222 6498 tablet_bootstrap.cc:492] T de6c7753c8564107b858876fcfa5924d P 04088ee3bea440a7bd5ff772152b9410: No bootstrap required, opened a new log
I20240501 14:05:20.168349 6498 ts_tablet_manager.cc:1367] T de6c7753c8564107b858876fcfa5924d P 04088ee3bea440a7bd5ff772152b9410: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240501 14:05:20.168627 6502 tablet_bootstrap.cc:492] T de6c7753c8564107b858876fcfa5924d P be56773105e744919601127a94e98adf: No bootstrap required, opened a new log
I20240501 14:05:20.168751 6502 ts_tablet_manager.cc:1367] T de6c7753c8564107b858876fcfa5924d P be56773105e744919601127a94e98adf: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240501 14:05:20.168710 6497 raft_consensus.cc:348] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:20.168905 6497 raft_consensus.cc:374] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:20.168996 6497 raft_consensus.cc:729] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c3fdc5272484ac09ef3e5e36a08f27c, State: Initialized, Role: FOLLOWER
I20240501 14:05:20.169014 6498 raft_consensus.cc:348] T de6c7753c8564107b858876fcfa5924d P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:20.169198 6498 raft_consensus.cc:374] T de6c7753c8564107b858876fcfa5924d P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:20.169323 6498 raft_consensus.cc:729] T de6c7753c8564107b858876fcfa5924d P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04088ee3bea440a7bd5ff772152b9410, State: Initialized, Role: FOLLOWER
I20240501 14:05:20.169283 6497 consensus_queue.cc:260] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:20.169569 6498 consensus_queue.cc:260] T de6c7753c8564107b858876fcfa5924d P 04088ee3bea440a7bd5ff772152b9410 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:20.169559 6502 raft_consensus.cc:348] T de6c7753c8564107b858876fcfa5924d P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:20.169744 6502 raft_consensus.cc:374] T de6c7753c8564107b858876fcfa5924d P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:20.169822 6502 raft_consensus.cc:729] T de6c7753c8564107b858876fcfa5924d P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be56773105e744919601127a94e98adf, State: Initialized, Role: FOLLOWER
I20240501 14:05:20.169838 6497 ts_tablet_manager.cc:1398] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:05:20.170024 6502 consensus_queue.cc:260] T de6c7753c8564107b858876fcfa5924d P be56773105e744919601127a94e98adf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:20.170161 6498 ts_tablet_manager.cc:1398] T de6c7753c8564107b858876fcfa5924d P 04088ee3bea440a7bd5ff772152b9410: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:05:20.170537 6502 ts_tablet_manager.cc:1398] T de6c7753c8564107b858876fcfa5924d P be56773105e744919601127a94e98adf: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:05:20.171213 6501 tablet_bootstrap.cc:492] T 8cc5eec4600640f7a293877c825f8d96 P e9c60449cf3c46f8aca4ab6c535302a4: No bootstrap required, opened a new log
I20240501 14:05:20.171351 6501 ts_tablet_manager.cc:1367] T 8cc5eec4600640f7a293877c825f8d96 P e9c60449cf3c46f8aca4ab6c535302a4: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240501 14:05:20.172102 6501 raft_consensus.cc:348] T 8cc5eec4600640f7a293877c825f8d96 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } }
I20240501 14:05:20.172309 6501 raft_consensus.cc:374] T 8cc5eec4600640f7a293877c825f8d96 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:20.172407 6501 raft_consensus.cc:729] T 8cc5eec4600640f7a293877c825f8d96 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9c60449cf3c46f8aca4ab6c535302a4, State: Initialized, Role: FOLLOWER
I20240501 14:05:20.172626 6501 consensus_queue.cc:260] T 8cc5eec4600640f7a293877c825f8d96 P e9c60449cf3c46f8aca4ab6c535302a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } }
I20240501 14:05:20.173197 6501 ts_tablet_manager.cc:1398] T 8cc5eec4600640f7a293877c825f8d96 P e9c60449cf3c46f8aca4ab6c535302a4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:05:20.173508 6501 tablet_bootstrap.cc:492] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4: Bootstrap starting.
I20240501 14:05:20.174863 6501 tablet_bootstrap.cc:654] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:20.175635 6502 tablet_bootstrap.cc:492] T ac4b234f794149e4b6def7a3b358942d P be56773105e744919601127a94e98adf: Bootstrap starting.
I20240501 14:05:20.177199 6502 tablet_bootstrap.cc:654] T ac4b234f794149e4b6def7a3b358942d P be56773105e744919601127a94e98adf: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:20.179997 6502 tablet_bootstrap.cc:492] T ac4b234f794149e4b6def7a3b358942d P be56773105e744919601127a94e98adf: No bootstrap required, opened a new log
I20240501 14:05:20.180132 6502 ts_tablet_manager.cc:1367] T ac4b234f794149e4b6def7a3b358942d P be56773105e744919601127a94e98adf: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240501 14:05:20.180215 6501 tablet_bootstrap.cc:492] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4: No bootstrap required, opened a new log
I20240501 14:05:20.180362 6501 ts_tablet_manager.cc:1367] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.006s
I20240501 14:05:20.180714 6502 raft_consensus.cc:348] T ac4b234f794149e4b6def7a3b358942d P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } }
I20240501 14:05:20.180871 6502 raft_consensus.cc:374] T ac4b234f794149e4b6def7a3b358942d P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:20.180938 6502 raft_consensus.cc:729] T ac4b234f794149e4b6def7a3b358942d P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be56773105e744919601127a94e98adf, State: Initialized, Role: FOLLOWER
I20240501 14:05:20.180964 6501 raft_consensus.cc:348] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } }
I20240501 14:05:20.181171 6501 raft_consensus.cc:374] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:20.181242 6501 raft_consensus.cc:729] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9c60449cf3c46f8aca4ab6c535302a4, State: Initialized, Role: FOLLOWER
I20240501 14:05:20.181164 6502 consensus_queue.cc:260] T ac4b234f794149e4b6def7a3b358942d P be56773105e744919601127a94e98adf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } }
I20240501 14:05:20.181507 6501 consensus_queue.cc:260] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } }
I20240501 14:05:20.181684 6502 ts_tablet_manager.cc:1398] T ac4b234f794149e4b6def7a3b358942d P be56773105e744919601127a94e98adf: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 14:05:20.182018 6502 tablet_bootstrap.cc:492] T 8cc5eec4600640f7a293877c825f8d96 P be56773105e744919601127a94e98adf: Bootstrap starting.
I20240501 14:05:20.182119 6501 ts_tablet_manager.cc:1398] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:05:20.183216 6502 tablet_bootstrap.cc:654] T 8cc5eec4600640f7a293877c825f8d96 P be56773105e744919601127a94e98adf: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:20.186268 6502 tablet_bootstrap.cc:492] T 8cc5eec4600640f7a293877c825f8d96 P be56773105e744919601127a94e98adf: No bootstrap required, opened a new log
I20240501 14:05:20.186421 6502 ts_tablet_manager.cc:1367] T 8cc5eec4600640f7a293877c825f8d96 P be56773105e744919601127a94e98adf: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.003s
I20240501 14:05:20.187222 6502 raft_consensus.cc:348] T 8cc5eec4600640f7a293877c825f8d96 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } }
I20240501 14:05:20.187364 6502 raft_consensus.cc:374] T 8cc5eec4600640f7a293877c825f8d96 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:20.187425 6502 raft_consensus.cc:729] T 8cc5eec4600640f7a293877c825f8d96 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be56773105e744919601127a94e98adf, State: Initialized, Role: FOLLOWER
I20240501 14:05:20.187572 6502 consensus_queue.cc:260] T 8cc5eec4600640f7a293877c825f8d96 P be56773105e744919601127a94e98adf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } }
I20240501 14:05:20.187969 6502 ts_tablet_manager.cc:1398] T 8cc5eec4600640f7a293877c825f8d96 P be56773105e744919601127a94e98adf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:05:20.227579 6296 raft_consensus.cc:482] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:05:20.227779 6296 raft_consensus.cc:504] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } }
I20240501 14:05:20.228482 6296 leader_election.cc:288] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be56773105e744919601127a94e98adf (127.29.124.132:35459), 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129:44033)
I20240501 14:05:20.228943 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac4b234f794149e4b6def7a3b358942d" candidate_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04088ee3bea440a7bd5ff772152b9410" is_pre_election: true
I20240501 14:05:20.228947 5385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac4b234f794149e4b6def7a3b358942d" candidate_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf" is_pre_election: true
I20240501 14:05:20.229152 4995 raft_consensus.cc:2454] T ac4b234f794149e4b6def7a3b358942d P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9c60449cf3c46f8aca4ab6c535302a4 in term 0.
I20240501 14:05:20.229154 5385 raft_consensus.cc:2454] T ac4b234f794149e4b6def7a3b358942d P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9c60449cf3c46f8aca4ab6c535302a4 in term 0.
I20240501 14:05:20.229617 5060 leader_election.cc:302] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be56773105e744919601127a94e98adf, e9c60449cf3c46f8aca4ab6c535302a4; no voters:
I20240501 14:05:20.229904 6296 raft_consensus.cc:2781] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:05:20.229987 6296 raft_consensus.cc:482] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:05:20.230074 6296 raft_consensus.cc:3037] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:20.231854 6296 raft_consensus.cc:504] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } }
I20240501 14:05:20.232367 6296 leader_election.cc:288] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 [CANDIDATE]: Term 1 election: Requested vote from peers be56773105e744919601127a94e98adf (127.29.124.132:35459), 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129:44033)
I20240501 14:05:20.232767 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac4b234f794149e4b6def7a3b358942d" candidate_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04088ee3bea440a7bd5ff772152b9410"
I20240501 14:05:20.232770 5385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac4b234f794149e4b6def7a3b358942d" candidate_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf"
I20240501 14:05:20.232918 5385 raft_consensus.cc:3037] T ac4b234f794149e4b6def7a3b358942d P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:20.232933 4995 raft_consensus.cc:3037] T ac4b234f794149e4b6def7a3b358942d P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:20.234791 5385 raft_consensus.cc:2454] T ac4b234f794149e4b6def7a3b358942d P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9c60449cf3c46f8aca4ab6c535302a4 in term 1.
I20240501 14:05:20.235075 4995 raft_consensus.cc:2454] T ac4b234f794149e4b6def7a3b358942d P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9c60449cf3c46f8aca4ab6c535302a4 in term 1.
I20240501 14:05:20.235168 5060 leader_election.cc:302] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be56773105e744919601127a94e98adf, e9c60449cf3c46f8aca4ab6c535302a4; no voters:
I20240501 14:05:20.235443 6296 raft_consensus.cc:2781] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:05:20.235605 6296 raft_consensus.cc:686] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 LEADER]: Becoming Leader. State: Replica: e9c60449cf3c46f8aca4ab6c535302a4, State: Running, Role: LEADER
I20240501 14:05:20.235854 6296 consensus_queue.cc:237] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } }
I20240501 14:05:20.238463 4851 catalog_manager.cc:5484] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130). New cstate: current_term: 1 leader_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } health_report { overall_health: UNKNOWN } } }
I20240501 14:05:20.428179 6452 raft_consensus.cc:482] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:05:20.428380 6452 raft_consensus.cc:504] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:20.429134 6452 leader_election.cc:288] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129:44033), be56773105e744919601127a94e98adf (127.29.124.132:35459)
I20240501 14:05:20.429776 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de6c7753c8564107b858876fcfa5924d" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04088ee3bea440a7bd5ff772152b9410" is_pre_election: true
I20240501 14:05:20.429937 5385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de6c7753c8564107b858876fcfa5924d" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf" is_pre_election: true
I20240501 14:05:20.430018 4995 raft_consensus.cc:2454] T de6c7753c8564107b858876fcfa5924d P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 0.
I20240501 14:05:20.430187 5385 raft_consensus.cc:2454] T de6c7753c8564107b858876fcfa5924d P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 0.
I20240501 14:05:20.430441 5190 leader_election.cc:302] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04088ee3bea440a7bd5ff772152b9410, 1c3fdc5272484ac09ef3e5e36a08f27c; no voters:
I20240501 14:05:20.430724 6452 raft_consensus.cc:2781] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:05:20.430845 6452 raft_consensus.cc:482] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:05:20.430929 6452 raft_consensus.cc:3037] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:20.432701 6452 raft_consensus.cc:504] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:20.433245 6452 leader_election.cc:288] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 1 election: Requested vote from peers 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129:44033), be56773105e744919601127a94e98adf (127.29.124.132:35459)
I20240501 14:05:20.433756 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de6c7753c8564107b858876fcfa5924d" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04088ee3bea440a7bd5ff772152b9410"
I20240501 14:05:20.433806 5385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de6c7753c8564107b858876fcfa5924d" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf"
I20240501 14:05:20.433970 4995 raft_consensus.cc:3037] T de6c7753c8564107b858876fcfa5924d P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:20.434005 5385 raft_consensus.cc:3037] T de6c7753c8564107b858876fcfa5924d P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:20.436650 5385 raft_consensus.cc:2454] T de6c7753c8564107b858876fcfa5924d P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 1.
I20240501 14:05:20.436662 4995 raft_consensus.cc:2454] T de6c7753c8564107b858876fcfa5924d P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 1.
I20240501 14:05:20.437029 5190 leader_election.cc:302] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters:
I20240501 14:05:20.437304 6452 raft_consensus.cc:2781] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:05:20.437479 6452 raft_consensus.cc:686] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEADER]: Becoming Leader. State: Replica: 1c3fdc5272484ac09ef3e5e36a08f27c, State: Running, Role: LEADER
I20240501 14:05:20.437793 6452 consensus_queue.cc:237] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:20.440675 4851 catalog_manager.cc:5484] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131). New cstate: current_term: 1 leader_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } health_report { overall_health: UNKNOWN } } }
I20240501 14:05:20.563659 6459 raft_consensus.cc:482] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:05:20.563859 6459 raft_consensus.cc:504] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } }
I20240501 14:05:20.564559 6459 leader_election.cc:288] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers be56773105e744919601127a94e98adf (127.29.124.132:35459), e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377)
I20240501 14:05:20.565107 5385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cc5eec4600640f7a293877c825f8d96" candidate_uuid: "04088ee3bea440a7bd5ff772152b9410" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf" is_pre_election: true
I20240501 14:05:20.565140 5126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cc5eec4600640f7a293877c825f8d96" candidate_uuid: "04088ee3bea440a7bd5ff772152b9410" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" is_pre_election: true
I20240501 14:05:20.565480 5385 raft_consensus.cc:2454] T 8cc5eec4600640f7a293877c825f8d96 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04088ee3bea440a7bd5ff772152b9410 in term 0.
I20240501 14:05:20.565475 5126 raft_consensus.cc:2454] T 8cc5eec4600640f7a293877c825f8d96 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04088ee3bea440a7bd5ff772152b9410 in term 0.
I20240501 14:05:20.565914 4929 leader_election.cc:302] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04088ee3bea440a7bd5ff772152b9410, be56773105e744919601127a94e98adf; no voters:
I20240501 14:05:20.566172 6459 raft_consensus.cc:2781] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:05:20.566293 6459 raft_consensus.cc:482] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:05:20.566380 6459 raft_consensus.cc:3037] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:20.569092 6459 raft_consensus.cc:504] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } }
I20240501 14:05:20.569768 6459 leader_election.cc:288] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 [CANDIDATE]: Term 1 election: Requested vote from peers be56773105e744919601127a94e98adf (127.29.124.132:35459), e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377)
I20240501 14:05:20.570262 5385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cc5eec4600640f7a293877c825f8d96" candidate_uuid: "04088ee3bea440a7bd5ff772152b9410" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf"
I20240501 14:05:20.570472 5385 raft_consensus.cc:3037] T 8cc5eec4600640f7a293877c825f8d96 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:20.570564 5126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8cc5eec4600640f7a293877c825f8d96" candidate_uuid: "04088ee3bea440a7bd5ff772152b9410" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9c60449cf3c46f8aca4ab6c535302a4"
I20240501 14:05:20.570777 5126 raft_consensus.cc:3037] T 8cc5eec4600640f7a293877c825f8d96 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:20.572563 5126 raft_consensus.cc:2454] T 8cc5eec4600640f7a293877c825f8d96 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04088ee3bea440a7bd5ff772152b9410 in term 1.
I20240501 14:05:20.572952 4929 leader_election.cc:302] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04088ee3bea440a7bd5ff772152b9410, e9c60449cf3c46f8aca4ab6c535302a4; no voters:
I20240501 14:05:20.573309 5385 raft_consensus.cc:2454] T 8cc5eec4600640f7a293877c825f8d96 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04088ee3bea440a7bd5ff772152b9410 in term 1.
I20240501 14:05:20.573344 6459 raft_consensus.cc:2781] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:05:20.574097 6459 raft_consensus.cc:686] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEADER]: Becoming Leader. State: Replica: 04088ee3bea440a7bd5ff772152b9410, State: Running, Role: LEADER
I20240501 14:05:20.574621 6459 consensus_queue.cc:237] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } }
I20240501 14:05:20.578325 4851 catalog_manager.cc:5484] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 reported cstate change: term changed from 0 to 1, leader changed from <none> to 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129). New cstate: current_term: 1 leader_uuid: "04088ee3bea440a7bd5ff772152b9410" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: UNKNOWN } } }
I20240501 14:05:20.688114 6296 consensus_queue.cc:1035] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:05:20.693238 6296 consensus_queue.cc:1035] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:05:20.815411 6452 consensus_queue.cc:1035] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:05:20.821609 6448 consensus_queue.cc:1035] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:05:21.023854 6518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:05:21.024171 6518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:05:21.031436 6518 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:05:21.033813 6459 consensus_queue.cc:1035] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:05:21.040129 6512 consensus_queue.cc:1035] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_6'
INSERT report
rows total: 2000
time total: 214.264 ms
time per row: 0.107132 ms
I20240501 14:05:21.387264 4849 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55620:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:05:21.395669 5234 tablet_service.cc:1467] Processing CreateTablet for tablet 63650547ab444f45bb377bcb49f258ad (DEFAULT_TABLE table=rebalance_test_table_7 [id=763c884aecd640b49c65c692fbde9422]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:21.395828 4973 tablet_service.cc:1467] Processing CreateTablet for tablet 63650547ab444f45bb377bcb49f258ad (DEFAULT_TABLE table=rebalance_test_table_7 [id=763c884aecd640b49c65c692fbde9422]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:21.396109 5234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63650547ab444f45bb377bcb49f258ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:21.396209 4973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63650547ab444f45bb377bcb49f258ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:21.396360 4974 tablet_service.cc:1467] Processing CreateTablet for tablet 71bdb382b08341218c8bf2459fe65fc0 (DEFAULT_TABLE table=rebalance_test_table_7 [id=763c884aecd640b49c65c692fbde9422]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:21.396736 4974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71bdb382b08341218c8bf2459fe65fc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:21.396945 5105 tablet_service.cc:1467] Processing CreateTablet for tablet 63650547ab444f45bb377bcb49f258ad (DEFAULT_TABLE table=rebalance_test_table_7 [id=763c884aecd640b49c65c692fbde9422]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:21.397358 5105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63650547ab444f45bb377bcb49f258ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:21.402400 5364 tablet_service.cc:1467] Processing CreateTablet for tablet 23ca2bf1e59f44fcac8a48aff8d056b1 (DEFAULT_TABLE table=rebalance_test_table_7 [id=763c884aecd640b49c65c692fbde9422]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:21.402820 5364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23ca2bf1e59f44fcac8a48aff8d056b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:21.402880 5106 tablet_service.cc:1467] Processing CreateTablet for tablet 23ca2bf1e59f44fcac8a48aff8d056b1 (DEFAULT_TABLE table=rebalance_test_table_7 [id=763c884aecd640b49c65c692fbde9422]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:21.403314 5106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23ca2bf1e59f44fcac8a48aff8d056b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:21.403259 5365 tablet_service.cc:1467] Processing CreateTablet for tablet 71bdb382b08341218c8bf2459fe65fc0 (DEFAULT_TABLE table=rebalance_test_table_7 [id=763c884aecd640b49c65c692fbde9422]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:21.403614 5365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71bdb382b08341218c8bf2459fe65fc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:21.406076 5236 tablet_service.cc:1467] Processing CreateTablet for tablet 71bdb382b08341218c8bf2459fe65fc0 (DEFAULT_TABLE table=rebalance_test_table_7 [id=763c884aecd640b49c65c692fbde9422]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:21.406482 5236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71bdb382b08341218c8bf2459fe65fc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:21.409652 5235 tablet_service.cc:1467] Processing CreateTablet for tablet 23ca2bf1e59f44fcac8a48aff8d056b1 (DEFAULT_TABLE table=rebalance_test_table_7 [id=763c884aecd640b49c65c692fbde9422]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:21.410039 5235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23ca2bf1e59f44fcac8a48aff8d056b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:21.412616 6575 tablet_bootstrap.cc:492] T 23ca2bf1e59f44fcac8a48aff8d056b1 P e9c60449cf3c46f8aca4ab6c535302a4: Bootstrap starting.
I20240501 14:05:21.414873 6575 tablet_bootstrap.cc:654] T 23ca2bf1e59f44fcac8a48aff8d056b1 P e9c60449cf3c46f8aca4ab6c535302a4: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:21.415256 6576 tablet_bootstrap.cc:492] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf: Bootstrap starting.
I20240501 14:05:21.416970 6576 tablet_bootstrap.cc:654] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:21.423568 6575 tablet_bootstrap.cc:492] T 23ca2bf1e59f44fcac8a48aff8d056b1 P e9c60449cf3c46f8aca4ab6c535302a4: No bootstrap required, opened a new log
I20240501 14:05:21.423820 6575 ts_tablet_manager.cc:1367] T 23ca2bf1e59f44fcac8a48aff8d056b1 P e9c60449cf3c46f8aca4ab6c535302a4: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20240501 14:05:21.425200 6577 tablet_bootstrap.cc:492] T 71bdb382b08341218c8bf2459fe65fc0 P 04088ee3bea440a7bd5ff772152b9410: Bootstrap starting.
I20240501 14:05:21.424666 6575 raft_consensus.cc:348] T 23ca2bf1e59f44fcac8a48aff8d056b1 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:21.425490 6575 raft_consensus.cc:374] T 23ca2bf1e59f44fcac8a48aff8d056b1 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:21.425683 6575 raft_consensus.cc:729] T 23ca2bf1e59f44fcac8a48aff8d056b1 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9c60449cf3c46f8aca4ab6c535302a4, State: Initialized, Role: FOLLOWER
I20240501 14:05:21.425941 6575 consensus_queue.cc:260] T 23ca2bf1e59f44fcac8a48aff8d056b1 P e9c60449cf3c46f8aca4ab6c535302a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:21.426707 6575 ts_tablet_manager.cc:1398] T 23ca2bf1e59f44fcac8a48aff8d056b1 P e9c60449cf3c46f8aca4ab6c535302a4: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:05:21.427000 6575 tablet_bootstrap.cc:492] T 63650547ab444f45bb377bcb49f258ad P e9c60449cf3c46f8aca4ab6c535302a4: Bootstrap starting.
I20240501 14:05:21.427289 6577 tablet_bootstrap.cc:654] T 71bdb382b08341218c8bf2459fe65fc0 P 04088ee3bea440a7bd5ff772152b9410: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:21.428133 6576 tablet_bootstrap.cc:492] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf: No bootstrap required, opened a new log
I20240501 14:05:21.428274 6576 ts_tablet_manager.cc:1367] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.001s
I20240501 14:05:21.428465 6575 tablet_bootstrap.cc:654] T 63650547ab444f45bb377bcb49f258ad P e9c60449cf3c46f8aca4ab6c535302a4: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:21.428856 6574 tablet_bootstrap.cc:492] T 63650547ab444f45bb377bcb49f258ad P 1c3fdc5272484ac09ef3e5e36a08f27c: Bootstrap starting.
I20240501 14:05:21.429005 6576 raft_consensus.cc:348] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:21.429193 6576 raft_consensus.cc:374] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:21.429299 6576 raft_consensus.cc:729] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be56773105e744919601127a94e98adf, State: Initialized, Role: FOLLOWER
I20240501 14:05:21.429558 6576 consensus_queue.cc:260] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:21.430528 6576 ts_tablet_manager.cc:1398] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:05:21.430707 6574 tablet_bootstrap.cc:654] T 63650547ab444f45bb377bcb49f258ad P 1c3fdc5272484ac09ef3e5e36a08f27c: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:21.430881 6576 tablet_bootstrap.cc:492] T 23ca2bf1e59f44fcac8a48aff8d056b1 P be56773105e744919601127a94e98adf: Bootstrap starting.
I20240501 14:05:21.432332 6576 tablet_bootstrap.cc:654] T 23ca2bf1e59f44fcac8a48aff8d056b1 P be56773105e744919601127a94e98adf: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:21.438326 6575 tablet_bootstrap.cc:492] T 63650547ab444f45bb377bcb49f258ad P e9c60449cf3c46f8aca4ab6c535302a4: No bootstrap required, opened a new log
I20240501 14:05:21.438436 6576 tablet_bootstrap.cc:492] T 23ca2bf1e59f44fcac8a48aff8d056b1 P be56773105e744919601127a94e98adf: No bootstrap required, opened a new log
I20240501 14:05:21.438622 6576 ts_tablet_manager.cc:1367] T 23ca2bf1e59f44fcac8a48aff8d056b1 P be56773105e744919601127a94e98adf: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20240501 14:05:21.438654 6577 tablet_bootstrap.cc:492] T 71bdb382b08341218c8bf2459fe65fc0 P 04088ee3bea440a7bd5ff772152b9410: No bootstrap required, opened a new log
I20240501 14:05:21.438735 6575 ts_tablet_manager.cc:1367] T 63650547ab444f45bb377bcb49f258ad P e9c60449cf3c46f8aca4ab6c535302a4: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.008s
I20240501 14:05:21.438774 6577 ts_tablet_manager.cc:1367] T 71bdb382b08341218c8bf2459fe65fc0 P 04088ee3bea440a7bd5ff772152b9410: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.005s
I20240501 14:05:21.439422 6576 raft_consensus.cc:348] T 23ca2bf1e59f44fcac8a48aff8d056b1 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:21.438668 6574 tablet_bootstrap.cc:492] T 63650547ab444f45bb377bcb49f258ad P 1c3fdc5272484ac09ef3e5e36a08f27c: No bootstrap required, opened a new log
I20240501 14:05:21.439637 6576 raft_consensus.cc:374] T 23ca2bf1e59f44fcac8a48aff8d056b1 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:21.439711 6576 raft_consensus.cc:729] T 23ca2bf1e59f44fcac8a48aff8d056b1 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be56773105e744919601127a94e98adf, State: Initialized, Role: FOLLOWER
I20240501 14:05:21.439522 6577 raft_consensus.cc:348] T 71bdb382b08341218c8bf2459fe65fc0 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:21.439817 6574 ts_tablet_manager.cc:1367] T 63650547ab444f45bb377bcb49f258ad P 1c3fdc5272484ac09ef3e5e36a08f27c: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.005s
I20240501 14:05:21.439832 6577 raft_consensus.cc:374] T 71bdb382b08341218c8bf2459fe65fc0 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:21.439743 6575 raft_consensus.cc:348] T 63650547ab444f45bb377bcb49f258ad P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } }
I20240501 14:05:21.439950 6575 raft_consensus.cc:374] T 63650547ab444f45bb377bcb49f258ad P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:21.439993 6577 raft_consensus.cc:729] T 71bdb382b08341218c8bf2459fe65fc0 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04088ee3bea440a7bd5ff772152b9410, State: Initialized, Role: FOLLOWER
I20240501 14:05:21.439967 6576 consensus_queue.cc:260] T 23ca2bf1e59f44fcac8a48aff8d056b1 P be56773105e744919601127a94e98adf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:21.440156 6575 raft_consensus.cc:729] T 63650547ab444f45bb377bcb49f258ad P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9c60449cf3c46f8aca4ab6c535302a4, State: Initialized, Role: FOLLOWER
I20240501 14:05:21.440467 6576 ts_tablet_manager.cc:1398] T 23ca2bf1e59f44fcac8a48aff8d056b1 P be56773105e744919601127a94e98adf: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:05:21.440876 6575 consensus_queue.cc:260] T 63650547ab444f45bb377bcb49f258ad P e9c60449cf3c46f8aca4ab6c535302a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } }
I20240501 14:05:21.440876 6577 consensus_queue.cc:260] T 71bdb382b08341218c8bf2459fe65fc0 P 04088ee3bea440a7bd5ff772152b9410 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:21.441824 6574 raft_consensus.cc:348] T 63650547ab444f45bb377bcb49f258ad P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } }
I20240501 14:05:21.442021 6574 raft_consensus.cc:374] T 63650547ab444f45bb377bcb49f258ad P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:21.442116 6574 raft_consensus.cc:729] T 63650547ab444f45bb377bcb49f258ad P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c3fdc5272484ac09ef3e5e36a08f27c, State: Initialized, Role: FOLLOWER
I20240501 14:05:21.442147 6577 ts_tablet_manager.cc:1398] T 71bdb382b08341218c8bf2459fe65fc0 P 04088ee3bea440a7bd5ff772152b9410: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:05:21.442379 6574 consensus_queue.cc:260] T 63650547ab444f45bb377bcb49f258ad P 1c3fdc5272484ac09ef3e5e36a08f27c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } }
I20240501 14:05:21.442941 6574 ts_tablet_manager.cc:1398] T 63650547ab444f45bb377bcb49f258ad P 1c3fdc5272484ac09ef3e5e36a08f27c: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240501 14:05:21.443090 6575 ts_tablet_manager.cc:1398] T 63650547ab444f45bb377bcb49f258ad P e9c60449cf3c46f8aca4ab6c535302a4: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 14:05:21.443221 6574 tablet_bootstrap.cc:492] T 71bdb382b08341218c8bf2459fe65fc0 P 1c3fdc5272484ac09ef3e5e36a08f27c: Bootstrap starting.
I20240501 14:05:21.442528 6577 tablet_bootstrap.cc:492] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410: Bootstrap starting.
I20240501 14:05:21.444749 6574 tablet_bootstrap.cc:654] T 71bdb382b08341218c8bf2459fe65fc0 P 1c3fdc5272484ac09ef3e5e36a08f27c: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:21.446141 6577 tablet_bootstrap.cc:654] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:21.449960 6574 tablet_bootstrap.cc:492] T 71bdb382b08341218c8bf2459fe65fc0 P 1c3fdc5272484ac09ef3e5e36a08f27c: No bootstrap required, opened a new log
I20240501 14:05:21.450129 6574 ts_tablet_manager.cc:1367] T 71bdb382b08341218c8bf2459fe65fc0 P 1c3fdc5272484ac09ef3e5e36a08f27c: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20240501 14:05:21.450963 6574 raft_consensus.cc:348] T 71bdb382b08341218c8bf2459fe65fc0 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:21.451187 6574 raft_consensus.cc:374] T 71bdb382b08341218c8bf2459fe65fc0 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:21.451246 6574 raft_consensus.cc:729] T 71bdb382b08341218c8bf2459fe65fc0 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c3fdc5272484ac09ef3e5e36a08f27c, State: Initialized, Role: FOLLOWER
I20240501 14:05:21.451443 6574 consensus_queue.cc:260] T 71bdb382b08341218c8bf2459fe65fc0 P 1c3fdc5272484ac09ef3e5e36a08f27c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:21.451975 6577 tablet_bootstrap.cc:492] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410: No bootstrap required, opened a new log
I20240501 14:05:21.451975 6574 ts_tablet_manager.cc:1398] T 71bdb382b08341218c8bf2459fe65fc0 P 1c3fdc5272484ac09ef3e5e36a08f27c: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240501 14:05:21.452133 6577 ts_tablet_manager.cc:1367] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.002s
I20240501 14:05:21.452265 6574 tablet_bootstrap.cc:492] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c: Bootstrap starting.
I20240501 14:05:21.452850 6577 raft_consensus.cc:348] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } }
I20240501 14:05:21.453075 6577 raft_consensus.cc:374] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:21.453153 6577 raft_consensus.cc:729] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04088ee3bea440a7bd5ff772152b9410, State: Initialized, Role: FOLLOWER
I20240501 14:05:21.453419 6577 consensus_queue.cc:260] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } }
I20240501 14:05:21.453774 6574 tablet_bootstrap.cc:654] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:21.454696 6577 ts_tablet_manager.cc:1398] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:05:21.456734 6574 tablet_bootstrap.cc:492] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c: No bootstrap required, opened a new log
I20240501 14:05:21.456857 6574 ts_tablet_manager.cc:1367] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240501 14:05:21.457482 6574 raft_consensus.cc:348] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:21.457657 6574 raft_consensus.cc:374] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:21.457741 6574 raft_consensus.cc:729] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c3fdc5272484ac09ef3e5e36a08f27c, State: Initialized, Role: FOLLOWER
I20240501 14:05:21.457965 6574 consensus_queue.cc:260] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:21.458359 6574 ts_tablet_manager.cc:1398] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20240501 14:05:21.493422 6512 raft_consensus.cc:482] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:05:21.493644 6512 raft_consensus.cc:504] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } }
I20240501 14:05:21.494361 6512 leader_election.cc:288] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209), e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377)
I20240501 14:05:21.494740 5256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63650547ab444f45bb377bcb49f258ad" candidate_uuid: "04088ee3bea440a7bd5ff772152b9410" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" is_pre_election: true
I20240501 14:05:21.494817 5126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63650547ab444f45bb377bcb49f258ad" candidate_uuid: "04088ee3bea440a7bd5ff772152b9410" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" is_pre_election: true
I20240501 14:05:21.495023 5126 raft_consensus.cc:2454] T 63650547ab444f45bb377bcb49f258ad P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04088ee3bea440a7bd5ff772152b9410 in term 0.
I20240501 14:05:21.494993 5256 raft_consensus.cc:2454] T 63650547ab444f45bb377bcb49f258ad P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04088ee3bea440a7bd5ff772152b9410 in term 0.
I20240501 14:05:21.495409 4929 leader_election.cc:302] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04088ee3bea440a7bd5ff772152b9410, e9c60449cf3c46f8aca4ab6c535302a4; no voters:
I20240501 14:05:21.495712 6512 raft_consensus.cc:2781] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:05:21.495826 6512 raft_consensus.cc:482] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:05:21.495932 6512 raft_consensus.cc:3037] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:21.497822 6512 raft_consensus.cc:504] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } }
I20240501 14:05:21.498339 6512 leader_election.cc:288] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 [CANDIDATE]: Term 1 election: Requested vote from peers 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209), e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377)
I20240501 14:05:21.498682 5256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63650547ab444f45bb377bcb49f258ad" candidate_uuid: "04088ee3bea440a7bd5ff772152b9410" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c"
I20240501 14:05:21.498687 5126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63650547ab444f45bb377bcb49f258ad" candidate_uuid: "04088ee3bea440a7bd5ff772152b9410" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9c60449cf3c46f8aca4ab6c535302a4"
I20240501 14:05:21.498837 5126 raft_consensus.cc:3037] T 63650547ab444f45bb377bcb49f258ad P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:21.498864 5256 raft_consensus.cc:3037] T 63650547ab444f45bb377bcb49f258ad P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:21.499351 6400 raft_consensus.cc:482] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:05:21.499514 6400 raft_consensus.cc:504] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:21.500183 6400 leader_election.cc:288] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129:44033), 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209)
I20240501 14:05:21.500784 5254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71bdb382b08341218c8bf2459fe65fc0" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" is_pre_election: true
I20240501 14:05:21.500808 4992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71bdb382b08341218c8bf2459fe65fc0" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04088ee3bea440a7bd5ff772152b9410" is_pre_election: true
I20240501 14:05:21.500982 4992 raft_consensus.cc:2454] T 71bdb382b08341218c8bf2459fe65fc0 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 0.
I20240501 14:05:21.500995 5126 raft_consensus.cc:2454] T 63650547ab444f45bb377bcb49f258ad P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04088ee3bea440a7bd5ff772152b9410 in term 1.
I20240501 14:05:21.500993 5254 raft_consensus.cc:2454] T 71bdb382b08341218c8bf2459fe65fc0 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 0.
I20240501 14:05:21.501283 5256 raft_consensus.cc:2454] T 63650547ab444f45bb377bcb49f258ad P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04088ee3bea440a7bd5ff772152b9410 in term 1.
I20240501 14:05:21.501395 4929 leader_election.cc:302] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04088ee3bea440a7bd5ff772152b9410, e9c60449cf3c46f8aca4ab6c535302a4; no voters:
I20240501 14:05:21.501427 5319 leader_election.cc:302] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters:
I20240501 14:05:21.501683 6512 raft_consensus.cc:2781] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:05:21.501821 6400 raft_consensus.cc:2781] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:05:21.501829 6512 raft_consensus.cc:686] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEADER]: Becoming Leader. State: Replica: 04088ee3bea440a7bd5ff772152b9410, State: Running, Role: LEADER
I20240501 14:05:21.501936 6400 raft_consensus.cc:482] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:05:21.502027 6400 raft_consensus.cc:3037] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:21.502067 6512 consensus_queue.cc:237] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } }
I20240501 14:05:21.504321 6400 raft_consensus.cc:504] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:21.504905 6400 leader_election.cc:288] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 1 election: Requested vote from peers 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129:44033), 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209)
I20240501 14:05:21.505220 4992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71bdb382b08341218c8bf2459fe65fc0" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04088ee3bea440a7bd5ff772152b9410"
I20240501 14:05:21.505436 4992 raft_consensus.cc:3037] T 71bdb382b08341218c8bf2459fe65fc0 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:21.505400 5256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "71bdb382b08341218c8bf2459fe65fc0" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c"
I20240501 14:05:21.505568 5256 raft_consensus.cc:3037] T 71bdb382b08341218c8bf2459fe65fc0 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:21.505542 4849 catalog_manager.cc:5484] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 reported cstate change: term changed from 0 to 1, leader changed from <none> to 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129). New cstate: current_term: 1 leader_uuid: "04088ee3bea440a7bd5ff772152b9410" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: UNKNOWN } } }
I20240501 14:05:21.507969 4992 raft_consensus.cc:2454] T 71bdb382b08341218c8bf2459fe65fc0 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 1.
I20240501 14:05:21.508041 5256 raft_consensus.cc:2454] T 71bdb382b08341218c8bf2459fe65fc0 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 1.
I20240501 14:05:21.508339 5320 leader_election.cc:302] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04088ee3bea440a7bd5ff772152b9410, be56773105e744919601127a94e98adf; no voters:
I20240501 14:05:21.508729 6400 raft_consensus.cc:2781] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:05:21.509158 6400 raft_consensus.cc:686] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf [term 1 LEADER]: Becoming Leader. State: Replica: be56773105e744919601127a94e98adf, State: Running, Role: LEADER
I20240501 14:05:21.509483 6400 consensus_queue.cc:237] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } }
I20240501 14:05:21.512665 4849 catalog_manager.cc:5484] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf reported cstate change: term changed from 0 to 1, leader changed from <none> to be56773105e744919601127a94e98adf (127.29.124.132). New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } health_report { overall_health: UNKNOWN } } }
I20240501 14:05:21.521590 6452 raft_consensus.cc:482] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:05:21.521715 6452 raft_consensus.cc:504] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:21.522313 6452 leader_election.cc:288] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377), be56773105e744919601127a94e98adf (127.29.124.132:35459)
I20240501 14:05:21.522681 5126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ca2bf1e59f44fcac8a48aff8d056b1" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" is_pre_election: true
I20240501 14:05:21.522737 5384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ca2bf1e59f44fcac8a48aff8d056b1" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf" is_pre_election: true
I20240501 14:05:21.522871 5126 raft_consensus.cc:2454] T 23ca2bf1e59f44fcac8a48aff8d056b1 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 0.
I20240501 14:05:21.522895 5384 raft_consensus.cc:2454] T 23ca2bf1e59f44fcac8a48aff8d056b1 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 0.
I20240501 14:05:21.523244 5190 leader_election.cc:302] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, e9c60449cf3c46f8aca4ab6c535302a4; no voters:
I20240501 14:05:21.523499 6452 raft_consensus.cc:2781] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:05:21.523602 6452 raft_consensus.cc:482] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:05:21.523687 6452 raft_consensus.cc:3037] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:21.525336 6452 raft_consensus.cc:504] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:21.525863 6452 leader_election.cc:288] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 1 election: Requested vote from peers e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377), be56773105e744919601127a94e98adf (127.29.124.132:35459)
I20240501 14:05:21.526214 5126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ca2bf1e59f44fcac8a48aff8d056b1" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9c60449cf3c46f8aca4ab6c535302a4"
I20240501 14:05:21.526265 5384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23ca2bf1e59f44fcac8a48aff8d056b1" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf"
I20240501 14:05:21.526383 5126 raft_consensus.cc:3037] T 23ca2bf1e59f44fcac8a48aff8d056b1 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:21.526433 5384 raft_consensus.cc:3037] T 23ca2bf1e59f44fcac8a48aff8d056b1 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:21.528132 5126 raft_consensus.cc:2454] T 23ca2bf1e59f44fcac8a48aff8d056b1 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 1.
I20240501 14:05:21.528208 5384 raft_consensus.cc:2454] T 23ca2bf1e59f44fcac8a48aff8d056b1 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 1.
I20240501 14:05:21.528476 5190 leader_election.cc:302] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, e9c60449cf3c46f8aca4ab6c535302a4; no voters:
I20240501 14:05:21.528782 6452 raft_consensus.cc:2781] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:05:21.528923 6452 raft_consensus.cc:686] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEADER]: Becoming Leader. State: Replica: 1c3fdc5272484ac09ef3e5e36a08f27c, State: Running, Role: LEADER
I20240501 14:05:21.529197 6452 consensus_queue.cc:237] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:21.531602 4849 catalog_manager.cc:5484] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131). New cstate: current_term: 1 leader_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } health_report { overall_health: UNKNOWN } } }
W20240501 14:05:21.870615 6591 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:05:21.870910 6591 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:05:21.878186 6591 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:05:21.903774 6459 consensus_queue.cc:1035] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_7'
I20240501 14:05:21.909891 6459 consensus_queue.cc:1035] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:05:21.940290 6400 consensus_queue.cc:1035] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:05:21.942587 6588 consensus_queue.cc:1035] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:05:22.045447 6452 consensus_queue.cc:1035] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:05:22.084750 5125 raft_consensus.cc:1261] T 23ca2bf1e59f44fcac8a48aff8d056b1 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240501 14:05:22.089318 6637 consensus_queue.cc:1035] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:05:22.163758 5190 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.124.132:35459, user_credentials={real_user=slave}} blocked reactor thread for 52393.5us
INSERT report
rows total: 2000
time total: 304.443 ms
time per row: 0.152222 ms
I20240501 14:05:22.334717 4845 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55620:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:05:22.343346 5106 tablet_service.cc:1467] Processing CreateTablet for tablet 3d6859d53fd747a5a5e94722626fa522 (DEFAULT_TABLE table=rebalance_test_table_8 [id=f26ba2f4e5654a5fa72021df5929200d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:22.343724 5234 tablet_service.cc:1467] Processing CreateTablet for tablet 3d6859d53fd747a5a5e94722626fa522 (DEFAULT_TABLE table=rebalance_test_table_8 [id=f26ba2f4e5654a5fa72021df5929200d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:22.343850 5106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d6859d53fd747a5a5e94722626fa522. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:22.344103 5234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d6859d53fd747a5a5e94722626fa522. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:22.344864 4973 tablet_service.cc:1467] Processing CreateTablet for tablet 9d32e0707d244f2c9bbcb57f62b09f31 (DEFAULT_TABLE table=rebalance_test_table_8 [id=f26ba2f4e5654a5fa72021df5929200d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:22.344883 5364 tablet_service.cc:1467] Processing CreateTablet for tablet 9d32e0707d244f2c9bbcb57f62b09f31 (DEFAULT_TABLE table=rebalance_test_table_8 [id=f26ba2f4e5654a5fa72021df5929200d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:22.345337 5364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d32e0707d244f2c9bbcb57f62b09f31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:22.345659 4974 tablet_service.cc:1467] Processing CreateTablet for tablet 3d6859d53fd747a5a5e94722626fa522 (DEFAULT_TABLE table=rebalance_test_table_8 [id=f26ba2f4e5654a5fa72021df5929200d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:22.345839 5105 tablet_service.cc:1467] Processing CreateTablet for tablet 9d32e0707d244f2c9bbcb57f62b09f31 (DEFAULT_TABLE table=rebalance_test_table_8 [id=f26ba2f4e5654a5fa72021df5929200d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:22.346190 5105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d32e0707d244f2c9bbcb57f62b09f31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:22.349884 5235 tablet_service.cc:1467] Processing CreateTablet for tablet 99d958617b7e4ef385b94f4c73f4dd84 (DEFAULT_TABLE table=rebalance_test_table_8 [id=f26ba2f4e5654a5fa72021df5929200d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:22.350304 5235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99d958617b7e4ef385b94f4c73f4dd84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:22.352015 4973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d32e0707d244f2c9bbcb57f62b09f31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:22.352322 4974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d6859d53fd747a5a5e94722626fa522. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:22.360999 4975 tablet_service.cc:1467] Processing CreateTablet for tablet 99d958617b7e4ef385b94f4c73f4dd84 (DEFAULT_TABLE table=rebalance_test_table_8 [id=f26ba2f4e5654a5fa72021df5929200d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:22.361424 4975 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99d958617b7e4ef385b94f4c73f4dd84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:22.361526 5365 tablet_service.cc:1467] Processing CreateTablet for tablet 99d958617b7e4ef385b94f4c73f4dd84 (DEFAULT_TABLE table=rebalance_test_table_8 [id=f26ba2f4e5654a5fa72021df5929200d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:22.361876 5365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99d958617b7e4ef385b94f4c73f4dd84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:22.372481 6666 tablet_bootstrap.cc:492] T 9d32e0707d244f2c9bbcb57f62b09f31 P 04088ee3bea440a7bd5ff772152b9410: Bootstrap starting.
I20240501 14:05:22.373245 6665 tablet_bootstrap.cc:492] T 99d958617b7e4ef385b94f4c73f4dd84 P 1c3fdc5272484ac09ef3e5e36a08f27c: Bootstrap starting.
I20240501 14:05:22.375492 6666 tablet_bootstrap.cc:654] T 9d32e0707d244f2c9bbcb57f62b09f31 P 04088ee3bea440a7bd5ff772152b9410: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:22.375519 6665 tablet_bootstrap.cc:654] T 99d958617b7e4ef385b94f4c73f4dd84 P 1c3fdc5272484ac09ef3e5e36a08f27c: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:22.377105 6664 tablet_bootstrap.cc:492] T 9d32e0707d244f2c9bbcb57f62b09f31 P be56773105e744919601127a94e98adf: Bootstrap starting.
I20240501 14:05:22.377347 6663 tablet_bootstrap.cc:492] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4: Bootstrap starting.
I20240501 14:05:22.378482 6666 tablet_bootstrap.cc:492] T 9d32e0707d244f2c9bbcb57f62b09f31 P 04088ee3bea440a7bd5ff772152b9410: No bootstrap required, opened a new log
I20240501 14:05:22.378631 6666 ts_tablet_manager.cc:1367] T 9d32e0707d244f2c9bbcb57f62b09f31 P 04088ee3bea440a7bd5ff772152b9410: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240501 14:05:22.378666 6663 tablet_bootstrap.cc:654] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:22.378722 6664 tablet_bootstrap.cc:654] T 9d32e0707d244f2c9bbcb57f62b09f31 P be56773105e744919601127a94e98adf: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:22.379436 6666 raft_consensus.cc:348] T 9d32e0707d244f2c9bbcb57f62b09f31 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } }
I20240501 14:05:22.379643 6666 raft_consensus.cc:374] T 9d32e0707d244f2c9bbcb57f62b09f31 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:22.379734 6666 raft_consensus.cc:729] T 9d32e0707d244f2c9bbcb57f62b09f31 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04088ee3bea440a7bd5ff772152b9410, State: Initialized, Role: FOLLOWER
I20240501 14:05:22.379966 6666 consensus_queue.cc:260] T 9d32e0707d244f2c9bbcb57f62b09f31 P 04088ee3bea440a7bd5ff772152b9410 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } }
I20240501 14:05:22.380184 6665 tablet_bootstrap.cc:492] T 99d958617b7e4ef385b94f4c73f4dd84 P 1c3fdc5272484ac09ef3e5e36a08f27c: No bootstrap required, opened a new log
I20240501 14:05:22.380317 6665 ts_tablet_manager.cc:1367] T 99d958617b7e4ef385b94f4c73f4dd84 P 1c3fdc5272484ac09ef3e5e36a08f27c: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20240501 14:05:22.380551 6666 ts_tablet_manager.cc:1398] T 9d32e0707d244f2c9bbcb57f62b09f31 P 04088ee3bea440a7bd5ff772152b9410: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:05:22.380910 6666 tablet_bootstrap.cc:492] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410: Bootstrap starting.
I20240501 14:05:22.380970 6665 raft_consensus.cc:348] T 99d958617b7e4ef385b94f4c73f4dd84 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:22.381150 6665 raft_consensus.cc:374] T 99d958617b7e4ef385b94f4c73f4dd84 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:22.381242 6665 raft_consensus.cc:729] T 99d958617b7e4ef385b94f4c73f4dd84 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c3fdc5272484ac09ef3e5e36a08f27c, State: Initialized, Role: FOLLOWER
I20240501 14:05:22.381490 6665 consensus_queue.cc:260] T 99d958617b7e4ef385b94f4c73f4dd84 P 1c3fdc5272484ac09ef3e5e36a08f27c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:22.381971 6665 ts_tablet_manager.cc:1398] T 99d958617b7e4ef385b94f4c73f4dd84 P 1c3fdc5272484ac09ef3e5e36a08f27c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:05:22.382432 6666 tablet_bootstrap.cc:654] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:22.382899 6665 tablet_bootstrap.cc:492] T 3d6859d53fd747a5a5e94722626fa522 P 1c3fdc5272484ac09ef3e5e36a08f27c: Bootstrap starting.
I20240501 14:05:22.384335 6665 tablet_bootstrap.cc:654] T 3d6859d53fd747a5a5e94722626fa522 P 1c3fdc5272484ac09ef3e5e36a08f27c: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:22.388322 6663 tablet_bootstrap.cc:492] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4: No bootstrap required, opened a new log
I20240501 14:05:22.388476 6663 ts_tablet_manager.cc:1367] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20240501 14:05:22.389243 6663 raft_consensus.cc:348] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } }
I20240501 14:05:22.389494 6663 raft_consensus.cc:374] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:22.389593 6663 raft_consensus.cc:729] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9c60449cf3c46f8aca4ab6c535302a4, State: Initialized, Role: FOLLOWER
I20240501 14:05:22.389824 6663 consensus_queue.cc:260] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } }
I20240501 14:05:22.390275 6666 tablet_bootstrap.cc:492] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410: No bootstrap required, opened a new log
I20240501 14:05:22.390338 6664 tablet_bootstrap.cc:492] T 9d32e0707d244f2c9bbcb57f62b09f31 P be56773105e744919601127a94e98adf: No bootstrap required, opened a new log
I20240501 14:05:22.390431 6666 ts_tablet_manager.cc:1367] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20240501 14:05:22.390467 6664 ts_tablet_manager.cc:1367] T 9d32e0707d244f2c9bbcb57f62b09f31 P be56773105e744919601127a94e98adf: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.004s
I20240501 14:05:22.390468 6663 ts_tablet_manager.cc:1398] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:05:22.390726 6663 tablet_bootstrap.cc:492] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4: Bootstrap starting.
I20240501 14:05:22.391132 6666 raft_consensus.cc:348] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:22.391177 6664 raft_consensus.cc:348] T 9d32e0707d244f2c9bbcb57f62b09f31 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } }
I20240501 14:05:22.391346 6664 raft_consensus.cc:374] T 9d32e0707d244f2c9bbcb57f62b09f31 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:22.391346 6666 raft_consensus.cc:374] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:22.391418 6666 raft_consensus.cc:729] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04088ee3bea440a7bd5ff772152b9410, State: Initialized, Role: FOLLOWER
I20240501 14:05:22.391423 6664 raft_consensus.cc:729] T 9d32e0707d244f2c9bbcb57f62b09f31 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be56773105e744919601127a94e98adf, State: Initialized, Role: FOLLOWER
I20240501 14:05:22.391667 6666 consensus_queue.cc:260] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:22.391651 6664 consensus_queue.cc:260] T 9d32e0707d244f2c9bbcb57f62b09f31 P be56773105e744919601127a94e98adf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } }
I20240501 14:05:22.392163 6666 ts_tablet_manager.cc:1398] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:05:22.394176 6663 tablet_bootstrap.cc:654] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:22.394279 6664 ts_tablet_manager.cc:1398] T 9d32e0707d244f2c9bbcb57f62b09f31 P be56773105e744919601127a94e98adf: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240501 14:05:22.394546 6664 tablet_bootstrap.cc:492] T 99d958617b7e4ef385b94f4c73f4dd84 P be56773105e744919601127a94e98adf: Bootstrap starting.
I20240501 14:05:22.395630 6666 tablet_bootstrap.cc:492] T 3d6859d53fd747a5a5e94722626fa522 P 04088ee3bea440a7bd5ff772152b9410: Bootstrap starting.
I20240501 14:05:22.397035 6666 tablet_bootstrap.cc:654] T 3d6859d53fd747a5a5e94722626fa522 P 04088ee3bea440a7bd5ff772152b9410: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:22.399258 6665 tablet_bootstrap.cc:492] T 3d6859d53fd747a5a5e94722626fa522 P 1c3fdc5272484ac09ef3e5e36a08f27c: No bootstrap required, opened a new log
I20240501 14:05:22.399428 6665 ts_tablet_manager.cc:1367] T 3d6859d53fd747a5a5e94722626fa522 P 1c3fdc5272484ac09ef3e5e36a08f27c: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.004s
I20240501 14:05:22.400154 6665 raft_consensus.cc:348] T 3d6859d53fd747a5a5e94722626fa522 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } }
I20240501 14:05:22.400369 6665 raft_consensus.cc:374] T 3d6859d53fd747a5a5e94722626fa522 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:22.400452 6665 raft_consensus.cc:729] T 3d6859d53fd747a5a5e94722626fa522 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c3fdc5272484ac09ef3e5e36a08f27c, State: Initialized, Role: FOLLOWER
I20240501 14:05:22.400672 6665 consensus_queue.cc:260] T 3d6859d53fd747a5a5e94722626fa522 P 1c3fdc5272484ac09ef3e5e36a08f27c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } }
I20240501 14:05:22.401008 6664 tablet_bootstrap.cc:654] T 99d958617b7e4ef385b94f4c73f4dd84 P be56773105e744919601127a94e98adf: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:22.401141 6665 ts_tablet_manager.cc:1398] T 3d6859d53fd747a5a5e94722626fa522 P 1c3fdc5272484ac09ef3e5e36a08f27c: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:05:22.402119 6663 tablet_bootstrap.cc:492] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4: No bootstrap required, opened a new log
I20240501 14:05:22.402257 6663 ts_tablet_manager.cc:1367] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20240501 14:05:22.402851 6663 raft_consensus.cc:348] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } }
I20240501 14:05:22.403064 6663 raft_consensus.cc:374] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:22.403142 6663 raft_consensus.cc:729] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9c60449cf3c46f8aca4ab6c535302a4, State: Initialized, Role: FOLLOWER
I20240501 14:05:22.403355 6663 consensus_queue.cc:260] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } }
I20240501 14:05:22.403920 6663 ts_tablet_manager.cc:1398] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:05:22.407084 6664 tablet_bootstrap.cc:492] T 99d958617b7e4ef385b94f4c73f4dd84 P be56773105e744919601127a94e98adf: No bootstrap required, opened a new log
I20240501 14:05:22.407176 6666 tablet_bootstrap.cc:492] T 3d6859d53fd747a5a5e94722626fa522 P 04088ee3bea440a7bd5ff772152b9410: No bootstrap required, opened a new log
I20240501 14:05:22.407215 6664 ts_tablet_manager.cc:1367] T 99d958617b7e4ef385b94f4c73f4dd84 P be56773105e744919601127a94e98adf: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20240501 14:05:22.407305 6666 ts_tablet_manager.cc:1367] T 3d6859d53fd747a5a5e94722626fa522 P 04088ee3bea440a7bd5ff772152b9410: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.009s
I20240501 14:05:22.407764 6664 raft_consensus.cc:348] T 99d958617b7e4ef385b94f4c73f4dd84 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:22.407820 6666 raft_consensus.cc:348] T 3d6859d53fd747a5a5e94722626fa522 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } }
I20240501 14:05:22.407927 6664 raft_consensus.cc:374] T 99d958617b7e4ef385b94f4c73f4dd84 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:22.407991 6664 raft_consensus.cc:729] T 99d958617b7e4ef385b94f4c73f4dd84 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be56773105e744919601127a94e98adf, State: Initialized, Role: FOLLOWER
I20240501 14:05:22.408002 6666 raft_consensus.cc:374] T 3d6859d53fd747a5a5e94722626fa522 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:22.408102 6666 raft_consensus.cc:729] T 3d6859d53fd747a5a5e94722626fa522 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04088ee3bea440a7bd5ff772152b9410, State: Initialized, Role: FOLLOWER
I20240501 14:05:22.408214 6664 consensus_queue.cc:260] T 99d958617b7e4ef385b94f4c73f4dd84 P be56773105e744919601127a94e98adf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:22.408305 6666 consensus_queue.cc:260] T 3d6859d53fd747a5a5e94722626fa522 P 04088ee3bea440a7bd5ff772152b9410 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } }
I20240501 14:05:22.408708 6664 ts_tablet_manager.cc:1398] T 99d958617b7e4ef385b94f4c73f4dd84 P be56773105e744919601127a94e98adf: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:05:22.408767 6666 ts_tablet_manager.cc:1398] T 3d6859d53fd747a5a5e94722626fa522 P 04088ee3bea440a7bd5ff772152b9410: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:05:22.459872 6647 raft_consensus.cc:482] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:05:22.460037 6647 raft_consensus.cc:504] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } }
I20240501 14:05:22.460667 6647 leader_election.cc:288] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209), 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129:44033)
I20240501 14:05:22.461053 5254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d6859d53fd747a5a5e94722626fa522" candidate_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" is_pre_election: true
I20240501 14:05:22.461172 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d6859d53fd747a5a5e94722626fa522" candidate_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04088ee3bea440a7bd5ff772152b9410" is_pre_election: true
I20240501 14:05:22.461305 5254 raft_consensus.cc:2454] T 3d6859d53fd747a5a5e94722626fa522 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9c60449cf3c46f8aca4ab6c535302a4 in term 0.
I20240501 14:05:22.461436 4995 raft_consensus.cc:2454] T 3d6859d53fd747a5a5e94722626fa522 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9c60449cf3c46f8aca4ab6c535302a4 in term 0.
I20240501 14:05:22.461704 5059 leader_election.cc:302] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, e9c60449cf3c46f8aca4ab6c535302a4; no voters:
I20240501 14:05:22.461987 6647 raft_consensus.cc:2781] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:05:22.462098 6647 raft_consensus.cc:482] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:05:22.462188 6647 raft_consensus.cc:3037] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:22.464102 6647 raft_consensus.cc:504] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } }
I20240501 14:05:22.464668 6647 leader_election.cc:288] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 [CANDIDATE]: Term 1 election: Requested vote from peers 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209), 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129:44033)
I20240501 14:05:22.465003 5254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d6859d53fd747a5a5e94722626fa522" candidate_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c"
I20240501 14:05:22.465132 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d6859d53fd747a5a5e94722626fa522" candidate_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04088ee3bea440a7bd5ff772152b9410"
I20240501 14:05:22.465200 5254 raft_consensus.cc:3037] T 3d6859d53fd747a5a5e94722626fa522 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:22.465353 4995 raft_consensus.cc:3037] T 3d6859d53fd747a5a5e94722626fa522 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:22.467051 5254 raft_consensus.cc:2454] T 3d6859d53fd747a5a5e94722626fa522 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9c60449cf3c46f8aca4ab6c535302a4 in term 1.
I20240501 14:05:22.467192 4995 raft_consensus.cc:2454] T 3d6859d53fd747a5a5e94722626fa522 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9c60449cf3c46f8aca4ab6c535302a4 in term 1.
I20240501 14:05:22.467398 5059 leader_election.cc:302] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, e9c60449cf3c46f8aca4ab6c535302a4; no voters:
I20240501 14:05:22.467732 6647 raft_consensus.cc:2781] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:05:22.467850 6647 raft_consensus.cc:686] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 LEADER]: Becoming Leader. State: Replica: e9c60449cf3c46f8aca4ab6c535302a4, State: Running, Role: LEADER
I20240501 14:05:22.468082 6647 consensus_queue.cc:237] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } }
I20240501 14:05:22.470893 4852 catalog_manager.cc:5484] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130). New cstate: current_term: 1 leader_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } health_report { overall_health: UNKNOWN } } }
I20240501 14:05:22.477815 6647 raft_consensus.cc:482] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:05:22.477958 6647 raft_consensus.cc:504] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } }
I20240501 14:05:22.478433 6647 leader_election.cc:288] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129:44033), be56773105e744919601127a94e98adf (127.29.124.132:35459)
I20240501 14:05:22.478893 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d32e0707d244f2c9bbcb57f62b09f31" candidate_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04088ee3bea440a7bd5ff772152b9410" is_pre_election: true
I20240501 14:05:22.478986 5385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d32e0707d244f2c9bbcb57f62b09f31" candidate_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf" is_pre_election: true
I20240501 14:05:22.479121 4995 raft_consensus.cc:2454] T 9d32e0707d244f2c9bbcb57f62b09f31 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9c60449cf3c46f8aca4ab6c535302a4 in term 0.
I20240501 14:05:22.479224 5385 raft_consensus.cc:2454] T 9d32e0707d244f2c9bbcb57f62b09f31 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9c60449cf3c46f8aca4ab6c535302a4 in term 0.
I20240501 14:05:22.479506 5060 leader_election.cc:302] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04088ee3bea440a7bd5ff772152b9410, e9c60449cf3c46f8aca4ab6c535302a4; no voters:
I20240501 14:05:22.479741 6647 raft_consensus.cc:2781] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:05:22.479882 6647 raft_consensus.cc:482] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:05:22.480072 6647 raft_consensus.cc:3037] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:22.481947 6647 raft_consensus.cc:504] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } }
I20240501 14:05:22.482412 6647 leader_election.cc:288] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 [CANDIDATE]: Term 1 election: Requested vote from peers 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129:44033), be56773105e744919601127a94e98adf (127.29.124.132:35459)
I20240501 14:05:22.482808 4995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d32e0707d244f2c9bbcb57f62b09f31" candidate_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04088ee3bea440a7bd5ff772152b9410"
I20240501 14:05:22.482905 5385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9d32e0707d244f2c9bbcb57f62b09f31" candidate_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf"
I20240501 14:05:22.482990 4995 raft_consensus.cc:3037] T 9d32e0707d244f2c9bbcb57f62b09f31 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:22.483096 5385 raft_consensus.cc:3037] T 9d32e0707d244f2c9bbcb57f62b09f31 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:22.485730 5385 raft_consensus.cc:2454] T 9d32e0707d244f2c9bbcb57f62b09f31 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9c60449cf3c46f8aca4ab6c535302a4 in term 1.
I20240501 14:05:22.485745 4995 raft_consensus.cc:2454] T 9d32e0707d244f2c9bbcb57f62b09f31 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9c60449cf3c46f8aca4ab6c535302a4 in term 1.
I20240501 14:05:22.486088 5060 leader_election.cc:302] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04088ee3bea440a7bd5ff772152b9410, e9c60449cf3c46f8aca4ab6c535302a4; no voters:
I20240501 14:05:22.486306 6647 raft_consensus.cc:2781] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:05:22.486445 6647 raft_consensus.cc:686] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 LEADER]: Becoming Leader. State: Replica: e9c60449cf3c46f8aca4ab6c535302a4, State: Running, Role: LEADER
I20240501 14:05:22.486683 6647 consensus_queue.cc:237] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } }
I20240501 14:05:22.489420 4852 catalog_manager.cc:5484] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 reported cstate change: term changed from 0 to 1, leader changed from <none> to e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130). New cstate: current_term: 1 leader_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: HEALTHY } } }
I20240501 14:05:22.501183 6635 raft_consensus.cc:482] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:05:22.501355 6635 raft_consensus.cc:504] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:22.501962 6635 leader_election.cc:288] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209), be56773105e744919601127a94e98adf (127.29.124.132:35459)
I20240501 14:05:22.502398 5254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99d958617b7e4ef385b94f4c73f4dd84" candidate_uuid: "04088ee3bea440a7bd5ff772152b9410" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" is_pre_election: true
I20240501 14:05:22.502462 5385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99d958617b7e4ef385b94f4c73f4dd84" candidate_uuid: "04088ee3bea440a7bd5ff772152b9410" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf" is_pre_election: true
I20240501 14:05:22.502599 5254 raft_consensus.cc:2454] T 99d958617b7e4ef385b94f4c73f4dd84 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04088ee3bea440a7bd5ff772152b9410 in term 0.
I20240501 14:05:22.502672 5385 raft_consensus.cc:2454] T 99d958617b7e4ef385b94f4c73f4dd84 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04088ee3bea440a7bd5ff772152b9410 in term 0.
I20240501 14:05:22.503005 4928 leader_election.cc:302] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04088ee3bea440a7bd5ff772152b9410, 1c3fdc5272484ac09ef3e5e36a08f27c; no voters:
I20240501 14:05:22.503317 6635 raft_consensus.cc:2781] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:05:22.503422 6635 raft_consensus.cc:482] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:05:22.503508 6635 raft_consensus.cc:3037] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:22.505590 6635 raft_consensus.cc:504] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:22.506076 6635 leader_election.cc:288] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 [CANDIDATE]: Term 1 election: Requested vote from peers 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209), be56773105e744919601127a94e98adf (127.29.124.132:35459)
I20240501 14:05:22.506489 5254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99d958617b7e4ef385b94f4c73f4dd84" candidate_uuid: "04088ee3bea440a7bd5ff772152b9410" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c"
I20240501 14:05:22.506503 5385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99d958617b7e4ef385b94f4c73f4dd84" candidate_uuid: "04088ee3bea440a7bd5ff772152b9410" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf"
I20240501 14:05:22.506654 5385 raft_consensus.cc:3037] T 99d958617b7e4ef385b94f4c73f4dd84 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:22.506669 5254 raft_consensus.cc:3037] T 99d958617b7e4ef385b94f4c73f4dd84 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:22.508440 5385 raft_consensus.cc:2454] T 99d958617b7e4ef385b94f4c73f4dd84 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04088ee3bea440a7bd5ff772152b9410 in term 1.
I20240501 14:05:22.508468 5254 raft_consensus.cc:2454] T 99d958617b7e4ef385b94f4c73f4dd84 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04088ee3bea440a7bd5ff772152b9410 in term 1.
I20240501 14:05:22.508848 4929 leader_election.cc:302] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04088ee3bea440a7bd5ff772152b9410, be56773105e744919601127a94e98adf; no voters:
I20240501 14:05:22.509143 6635 raft_consensus.cc:2781] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:05:22.509330 6635 raft_consensus.cc:686] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEADER]: Becoming Leader. State: Replica: 04088ee3bea440a7bd5ff772152b9410, State: Running, Role: LEADER
I20240501 14:05:22.509588 6635 consensus_queue.cc:237] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:22.512408 4852 catalog_manager.cc:5484] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 reported cstate change: term changed from 0 to 1, leader changed from <none> to 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129). New cstate: current_term: 1 leader_uuid: "04088ee3bea440a7bd5ff772152b9410" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } health_report { overall_health: UNKNOWN } } }
W20240501 14:05:22.884507 6679 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:05:22.884822 6679 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:05:22.892277 6679 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
I20240501 14:05:22.937242 6647 consensus_queue.cc:1035] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:05:22.975082 6647 consensus_queue.cc:1035] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:05:22.981134 6635 consensus_queue.cc:1035] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:05:22.986748 6647 consensus_queue.cc:1035] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:05:23.001190 6603 consensus_queue.cc:1035] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:05:23.008924 6647 consensus_queue.cc:1035] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 247.138 ms
time per row: 0.123569 ms
I20240501 14:05:23.276515 4847 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55620:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240501 14:05:23.285300 5106 tablet_service.cc:1467] Processing CreateTablet for tablet 61a274b24f53458f88c2cde4f8dbe4f4 (DEFAULT_TABLE table=rebalance_test_table_9 [id=5fc33bdfba464f65b68de5887d425ab7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:23.285431 5235 tablet_service.cc:1467] Processing CreateTablet for tablet 61a274b24f53458f88c2cde4f8dbe4f4 (DEFAULT_TABLE table=rebalance_test_table_9 [id=5fc33bdfba464f65b68de5887d425ab7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:23.285789 5106 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61a274b24f53458f88c2cde4f8dbe4f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:23.285789 5235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61a274b24f53458f88c2cde4f8dbe4f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:23.285974 5364 tablet_service.cc:1467] Processing CreateTablet for tablet 61a274b24f53458f88c2cde4f8dbe4f4 (DEFAULT_TABLE table=rebalance_test_table_9 [id=5fc33bdfba464f65b68de5887d425ab7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:23.286392 5364 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61a274b24f53458f88c2cde4f8dbe4f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:23.286624 5365 tablet_service.cc:1467] Processing CreateTablet for tablet 420642af520d45ffbc7f2d35f93e75a7 (DEFAULT_TABLE table=rebalance_test_table_9 [id=5fc33bdfba464f65b68de5887d425ab7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:23.286937 5365 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 420642af520d45ffbc7f2d35f93e75a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:23.293622 4973 tablet_service.cc:1467] Processing CreateTablet for tablet ec1e81011c6a46bc8a39aecf3a5e71e4 (DEFAULT_TABLE table=rebalance_test_table_9 [id=5fc33bdfba464f65b68de5887d425ab7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:23.293702 5234 tablet_service.cc:1467] Processing CreateTablet for tablet 420642af520d45ffbc7f2d35f93e75a7 (DEFAULT_TABLE table=rebalance_test_table_9 [id=5fc33bdfba464f65b68de5887d425ab7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:23.293864 5236 tablet_service.cc:1467] Processing CreateTablet for tablet ec1e81011c6a46bc8a39aecf3a5e71e4 (DEFAULT_TABLE table=rebalance_test_table_9 [id=5fc33bdfba464f65b68de5887d425ab7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:23.294080 4973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec1e81011c6a46bc8a39aecf3a5e71e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:23.294087 5234 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 420642af520d45ffbc7f2d35f93e75a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:23.294287 5236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec1e81011c6a46bc8a39aecf3a5e71e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:23.300498 5104 tablet_service.cc:1467] Processing CreateTablet for tablet ec1e81011c6a46bc8a39aecf3a5e71e4 (DEFAULT_TABLE table=rebalance_test_table_9 [id=5fc33bdfba464f65b68de5887d425ab7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:23.300480 5105 tablet_service.cc:1467] Processing CreateTablet for tablet 420642af520d45ffbc7f2d35f93e75a7 (DEFAULT_TABLE table=rebalance_test_table_9 [id=5fc33bdfba464f65b68de5887d425ab7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240501 14:05:23.300930 5104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec1e81011c6a46bc8a39aecf3a5e71e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:23.301137 5105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 420642af520d45ffbc7f2d35f93e75a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:23.306490 6738 tablet_bootstrap.cc:492] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf: Bootstrap starting.
I20240501 14:05:23.307785 6739 tablet_bootstrap.cc:492] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 04088ee3bea440a7bd5ff772152b9410: Bootstrap starting.
I20240501 14:05:23.308647 6738 tablet_bootstrap.cc:654] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:23.310194 6739 tablet_bootstrap.cc:654] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 04088ee3bea440a7bd5ff772152b9410: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:23.314868 6740 tablet_bootstrap.cc:492] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c: Bootstrap starting.
I20240501 14:05:23.316709 6740 tablet_bootstrap.cc:654] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:23.316743 6738 tablet_bootstrap.cc:492] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf: No bootstrap required, opened a new log
I20240501 14:05:23.317119 6738 ts_tablet_manager.cc:1367] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20240501 14:05:23.317884 6739 tablet_bootstrap.cc:492] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 04088ee3bea440a7bd5ff772152b9410: No bootstrap required, opened a new log
I20240501 14:05:23.317884 6738 raft_consensus.cc:348] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } }
I20240501 14:05:23.318066 6739 ts_tablet_manager.cc:1367] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 04088ee3bea440a7bd5ff772152b9410: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.008s
I20240501 14:05:23.318106 6738 raft_consensus.cc:374] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:23.318234 6738 raft_consensus.cc:729] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be56773105e744919601127a94e98adf, State: Initialized, Role: FOLLOWER
I20240501 14:05:23.318286 6737 tablet_bootstrap.cc:492] T 61a274b24f53458f88c2cde4f8dbe4f4 P e9c60449cf3c46f8aca4ab6c535302a4: Bootstrap starting.
I20240501 14:05:23.318477 6738 consensus_queue.cc:260] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } }
I20240501 14:05:23.318809 6739 raft_consensus.cc:348] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } }
I20240501 14:05:23.319047 6739 raft_consensus.cc:374] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:23.319180 6739 raft_consensus.cc:729] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04088ee3bea440a7bd5ff772152b9410, State: Initialized, Role: FOLLOWER
I20240501 14:05:23.319451 6739 consensus_queue.cc:260] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 04088ee3bea440a7bd5ff772152b9410 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } }
I20240501 14:05:23.320106 6739 ts_tablet_manager.cc:1398] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 04088ee3bea440a7bd5ff772152b9410: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:05:23.320225 6737 tablet_bootstrap.cc:654] T 61a274b24f53458f88c2cde4f8dbe4f4 P e9c60449cf3c46f8aca4ab6c535302a4: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:23.321578 6738 ts_tablet_manager.cc:1398] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:05:23.321969 6738 tablet_bootstrap.cc:492] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf: Bootstrap starting.
I20240501 14:05:23.323568 6738 tablet_bootstrap.cc:654] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:23.324849 6737 tablet_bootstrap.cc:492] T 61a274b24f53458f88c2cde4f8dbe4f4 P e9c60449cf3c46f8aca4ab6c535302a4: No bootstrap required, opened a new log
I20240501 14:05:23.324999 6737 ts_tablet_manager.cc:1367] T 61a274b24f53458f88c2cde4f8dbe4f4 P e9c60449cf3c46f8aca4ab6c535302a4: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240501 14:05:23.325666 6737 raft_consensus.cc:348] T 61a274b24f53458f88c2cde4f8dbe4f4 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:23.325834 6737 raft_consensus.cc:374] T 61a274b24f53458f88c2cde4f8dbe4f4 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:23.325919 6737 raft_consensus.cc:729] T 61a274b24f53458f88c2cde4f8dbe4f4 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9c60449cf3c46f8aca4ab6c535302a4, State: Initialized, Role: FOLLOWER
I20240501 14:05:23.326118 6737 consensus_queue.cc:260] T 61a274b24f53458f88c2cde4f8dbe4f4 P e9c60449cf3c46f8aca4ab6c535302a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:23.327010 6737 ts_tablet_manager.cc:1398] T 61a274b24f53458f88c2cde4f8dbe4f4 P e9c60449cf3c46f8aca4ab6c535302a4: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:05:23.327252 6740 tablet_bootstrap.cc:492] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c: No bootstrap required, opened a new log
I20240501 14:05:23.327407 6740 ts_tablet_manager.cc:1367] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.005s
I20240501 14:05:23.328205 6738 tablet_bootstrap.cc:492] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf: No bootstrap required, opened a new log
I20240501 14:05:23.328117 6740 raft_consensus.cc:348] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } }
I20240501 14:05:23.328336 6740 raft_consensus.cc:374] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:23.328426 6740 raft_consensus.cc:729] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c3fdc5272484ac09ef3e5e36a08f27c, State: Initialized, Role: FOLLOWER
I20240501 14:05:23.328526 6738 ts_tablet_manager.cc:1367] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20240501 14:05:23.328639 6740 consensus_queue.cc:260] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } }
I20240501 14:05:23.328831 6737 tablet_bootstrap.cc:492] T 420642af520d45ffbc7f2d35f93e75a7 P e9c60449cf3c46f8aca4ab6c535302a4: Bootstrap starting.
I20240501 14:05:23.329864 6740 ts_tablet_manager.cc:1398] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240501 14:05:23.330121 6740 tablet_bootstrap.cc:492] T 61a274b24f53458f88c2cde4f8dbe4f4 P 1c3fdc5272484ac09ef3e5e36a08f27c: Bootstrap starting.
I20240501 14:05:23.330281 6737 tablet_bootstrap.cc:654] T 420642af520d45ffbc7f2d35f93e75a7 P e9c60449cf3c46f8aca4ab6c535302a4: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:23.330595 6738 raft_consensus.cc:348] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:23.330860 6738 raft_consensus.cc:374] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:23.330969 6738 raft_consensus.cc:729] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be56773105e744919601127a94e98adf, State: Initialized, Role: FOLLOWER
I20240501 14:05:23.331264 6738 consensus_queue.cc:260] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:23.332266 6740 tablet_bootstrap.cc:654] T 61a274b24f53458f88c2cde4f8dbe4f4 P 1c3fdc5272484ac09ef3e5e36a08f27c: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:23.332278 6738 ts_tablet_manager.cc:1398] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20240501 14:05:23.333981 6740 tablet_bootstrap.cc:492] T 61a274b24f53458f88c2cde4f8dbe4f4 P 1c3fdc5272484ac09ef3e5e36a08f27c: No bootstrap required, opened a new log
I20240501 14:05:23.333947 6603 raft_consensus.cc:482] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:05:23.334122 6740 ts_tablet_manager.cc:1367] T 61a274b24f53458f88c2cde4f8dbe4f4 P 1c3fdc5272484ac09ef3e5e36a08f27c: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20240501 14:05:23.334112 6603 raft_consensus.cc:504] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } }
I20240501 14:05:23.334779 6603 leader_election.cc:288] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 04088ee3bea440a7bd5ff772152b9410 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209), e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377)
I20240501 14:05:23.334736 6740 raft_consensus.cc:348] T 61a274b24f53458f88c2cde4f8dbe4f4 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:23.334918 6740 raft_consensus.cc:374] T 61a274b24f53458f88c2cde4f8dbe4f4 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:23.334985 6740 raft_consensus.cc:729] T 61a274b24f53458f88c2cde4f8dbe4f4 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c3fdc5272484ac09ef3e5e36a08f27c, State: Initialized, Role: FOLLOWER
I20240501 14:05:23.335215 6740 consensus_queue.cc:260] T 61a274b24f53458f88c2cde4f8dbe4f4 P 1c3fdc5272484ac09ef3e5e36a08f27c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:23.335589 5126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec1e81011c6a46bc8a39aecf3a5e71e4" candidate_uuid: "04088ee3bea440a7bd5ff772152b9410" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" is_pre_election: true
W20240501 14:05:23.336453 4929 leader_election.cc:341] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 04088ee3bea440a7bd5ff772152b9410 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:05:23.337072 5255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec1e81011c6a46bc8a39aecf3a5e71e4" candidate_uuid: "04088ee3bea440a7bd5ff772152b9410" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" is_pre_election: true
I20240501 14:05:23.337644 6740 ts_tablet_manager.cc:1398] T 61a274b24f53458f88c2cde4f8dbe4f4 P 1c3fdc5272484ac09ef3e5e36a08f27c: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
W20240501 14:05:23.337688 4928 leader_election.cc:341] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 04088ee3bea440a7bd5ff772152b9410 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209): Illegal state: must be running to vote when last-logged opid is not known
I20240501 14:05:23.337814 4928 leader_election.cc:302] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 04088ee3bea440a7bd5ff772152b9410 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 04088ee3bea440a7bd5ff772152b9410; no voters: 1c3fdc5272484ac09ef3e5e36a08f27c, e9c60449cf3c46f8aca4ab6c535302a4
I20240501 14:05:23.337927 6740 tablet_bootstrap.cc:492] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c: Bootstrap starting.
I20240501 14:05:23.338238 6603 raft_consensus.cc:2726] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240501 14:05:23.338335 6737 tablet_bootstrap.cc:492] T 420642af520d45ffbc7f2d35f93e75a7 P e9c60449cf3c46f8aca4ab6c535302a4: No bootstrap required, opened a new log
I20240501 14:05:23.338472 6737 ts_tablet_manager.cc:1367] T 420642af520d45ffbc7f2d35f93e75a7 P e9c60449cf3c46f8aca4ab6c535302a4: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.003s
I20240501 14:05:23.339064 6737 raft_consensus.cc:348] T 420642af520d45ffbc7f2d35f93e75a7 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } }
I20240501 14:05:23.339161 6740 tablet_bootstrap.cc:654] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:23.339236 6737 raft_consensus.cc:374] T 420642af520d45ffbc7f2d35f93e75a7 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:23.339318 6737 raft_consensus.cc:729] T 420642af520d45ffbc7f2d35f93e75a7 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9c60449cf3c46f8aca4ab6c535302a4, State: Initialized, Role: FOLLOWER
I20240501 14:05:23.339542 6737 consensus_queue.cc:260] T 420642af520d45ffbc7f2d35f93e75a7 P e9c60449cf3c46f8aca4ab6c535302a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } }
I20240501 14:05:23.340147 6737 ts_tablet_manager.cc:1398] T 420642af520d45ffbc7f2d35f93e75a7 P e9c60449cf3c46f8aca4ab6c535302a4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240501 14:05:23.340452 6737 tablet_bootstrap.cc:492] T ec1e81011c6a46bc8a39aecf3a5e71e4 P e9c60449cf3c46f8aca4ab6c535302a4: Bootstrap starting.
I20240501 14:05:23.341454 6740 tablet_bootstrap.cc:492] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c: No bootstrap required, opened a new log
I20240501 14:05:23.341636 6740 ts_tablet_manager.cc:1367] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.002s
I20240501 14:05:23.341885 6737 tablet_bootstrap.cc:654] T ec1e81011c6a46bc8a39aecf3a5e71e4 P e9c60449cf3c46f8aca4ab6c535302a4: Neither blocks nor log segments found. Creating new log.
I20240501 14:05:23.342363 6740 raft_consensus.cc:348] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } }
I20240501 14:05:23.342823 6740 raft_consensus.cc:374] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:23.342911 6740 raft_consensus.cc:729] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1c3fdc5272484ac09ef3e5e36a08f27c, State: Initialized, Role: FOLLOWER
I20240501 14:05:23.343118 6740 consensus_queue.cc:260] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } }
I20240501 14:05:23.343680 6737 tablet_bootstrap.cc:492] T ec1e81011c6a46bc8a39aecf3a5e71e4 P e9c60449cf3c46f8aca4ab6c535302a4: No bootstrap required, opened a new log
I20240501 14:05:23.343670 6740 ts_tablet_manager.cc:1398] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:05:23.343830 6737 ts_tablet_manager.cc:1367] T ec1e81011c6a46bc8a39aecf3a5e71e4 P e9c60449cf3c46f8aca4ab6c535302a4: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.002s
I20240501 14:05:23.344447 6737 raft_consensus.cc:348] T ec1e81011c6a46bc8a39aecf3a5e71e4 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } }
I20240501 14:05:23.344592 6737 raft_consensus.cc:374] T ec1e81011c6a46bc8a39aecf3a5e71e4 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240501 14:05:23.344674 6737 raft_consensus.cc:729] T ec1e81011c6a46bc8a39aecf3a5e71e4 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9c60449cf3c46f8aca4ab6c535302a4, State: Initialized, Role: FOLLOWER
I20240501 14:05:23.344882 6737 consensus_queue.cc:260] T ec1e81011c6a46bc8a39aecf3a5e71e4 P e9c60449cf3c46f8aca4ab6c535302a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } }
I20240501 14:05:23.345448 6737 ts_tablet_manager.cc:1398] T ec1e81011c6a46bc8a39aecf3a5e71e4 P e9c60449cf3c46f8aca4ab6c535302a4: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20240501 14:05:23.376155 6588 raft_consensus.cc:482] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:05:23.376366 6588 raft_consensus.cc:504] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:23.377034 6588 leader_election.cc:288] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377), 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209)
I20240501 14:05:23.377472 5126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61a274b24f53458f88c2cde4f8dbe4f4" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" is_pre_election: true
I20240501 14:05:23.377511 5256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61a274b24f53458f88c2cde4f8dbe4f4" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" is_pre_election: true
I20240501 14:05:23.377725 5256 raft_consensus.cc:2454] T 61a274b24f53458f88c2cde4f8dbe4f4 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 0.
I20240501 14:05:23.377725 5126 raft_consensus.cc:2454] T 61a274b24f53458f88c2cde4f8dbe4f4 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 0.
I20240501 14:05:23.378136 5320 leader_election.cc:302] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be56773105e744919601127a94e98adf, e9c60449cf3c46f8aca4ab6c535302a4; no voters:
I20240501 14:05:23.378413 6588 raft_consensus.cc:2781] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:05:23.378516 6588 raft_consensus.cc:482] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:05:23.378623 6588 raft_consensus.cc:3037] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:23.380538 6588 raft_consensus.cc:504] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:23.381173 6588 leader_election.cc:288] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 1 election: Requested vote from peers e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377), 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209)
I20240501 14:05:23.381503 5126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61a274b24f53458f88c2cde4f8dbe4f4" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9c60449cf3c46f8aca4ab6c535302a4"
I20240501 14:05:23.381693 5126 raft_consensus.cc:3037] T 61a274b24f53458f88c2cde4f8dbe4f4 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:23.381672 5256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "61a274b24f53458f88c2cde4f8dbe4f4" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c"
I20240501 14:05:23.381882 5256 raft_consensus.cc:3037] T 61a274b24f53458f88c2cde4f8dbe4f4 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:23.382367 6588 raft_consensus.cc:482] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:05:23.382503 6588 raft_consensus.cc:504] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } }
I20240501 14:05:23.383210 6588 leader_election.cc:288] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209), e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377)
I20240501 14:05:23.383401 5255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420642af520d45ffbc7f2d35f93e75a7" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" is_pre_election: true
I20240501 14:05:23.383607 5255 raft_consensus.cc:2454] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 0.
I20240501 14:05:23.384083 5319 leader_election.cc:302] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters:
I20240501 14:05:23.384150 5125 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420642af520d45ffbc7f2d35f93e75a7" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" is_pre_election: true
I20240501 14:05:23.384220 5256 raft_consensus.cc:2454] T 61a274b24f53458f88c2cde4f8dbe4f4 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 1.
I20240501 14:05:23.384341 5125 raft_consensus.cc:2454] T 420642af520d45ffbc7f2d35f93e75a7 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 0.
I20240501 14:05:23.384377 6588 raft_consensus.cc:2781] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:05:23.384485 6588 raft_consensus.cc:482] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:05:23.384150 5126 raft_consensus.cc:2454] T 61a274b24f53458f88c2cde4f8dbe4f4 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 1.
I20240501 14:05:23.384645 5319 leader_election.cc:302] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters:
I20240501 14:05:23.384683 6588 raft_consensus.cc:3037] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:23.385421 6750 raft_consensus.cc:2781] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:05:23.385931 6750 raft_consensus.cc:686] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [term 1 LEADER]: Becoming Leader. State: Replica: be56773105e744919601127a94e98adf, State: Running, Role: LEADER
I20240501 14:05:23.386348 6750 consensus_queue.cc:237] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } }
I20240501 14:05:23.387521 6588 raft_consensus.cc:504] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } }
I20240501 14:05:23.388478 5256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420642af520d45ffbc7f2d35f93e75a7" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c"
I20240501 14:05:23.388495 5126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420642af520d45ffbc7f2d35f93e75a7" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9c60449cf3c46f8aca4ab6c535302a4"
I20240501 14:05:23.388648 5256 raft_consensus.cc:3037] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:23.388680 5126 raft_consensus.cc:3037] T 420642af520d45ffbc7f2d35f93e75a7 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:23.389421 6588 leader_election.cc:288] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 1 election: Requested vote from peers 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209), e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377)
I20240501 14:05:23.389756 4849 catalog_manager.cc:5484] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf reported cstate change: term changed from 0 to 1, leader changed from <none> to be56773105e744919601127a94e98adf (127.29.124.132). New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } health_report { overall_health: HEALTHY } } }
I20240501 14:05:23.390585 5256 raft_consensus.cc:2454] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 1.
I20240501 14:05:23.390980 5319 leader_election.cc:302] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters:
I20240501 14:05:23.391085 5126 raft_consensus.cc:2454] T 420642af520d45ffbc7f2d35f93e75a7 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 1.
I20240501 14:05:23.391270 6588 raft_consensus.cc:2781] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:05:23.391862 6588 raft_consensus.cc:686] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf [term 1 LEADER]: Becoming Leader. State: Replica: be56773105e744919601127a94e98adf, State: Running, Role: LEADER
I20240501 14:05:23.392102 6588 consensus_queue.cc:237] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } }
I20240501 14:05:23.395673 4849 catalog_manager.cc:5484] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf reported cstate change: term changed from 0 to 1, leader changed from <none> to be56773105e744919601127a94e98adf (127.29.124.132). New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: UNKNOWN } } }
I20240501 14:05:23.652899 6711 raft_consensus.cc:482] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:05:23.653079 6711 raft_consensus.cc:504] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } }
I20240501 14:05:23.653765 6711 leader_election.cc:288] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129:44033), e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377)
I20240501 14:05:23.654280 5126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec1e81011c6a46bc8a39aecf3a5e71e4" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" is_pre_election: true
I20240501 14:05:23.654489 5126 raft_consensus.cc:2454] T ec1e81011c6a46bc8a39aecf3a5e71e4 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 0.
I20240501 14:05:23.654836 4992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec1e81011c6a46bc8a39aecf3a5e71e4" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04088ee3bea440a7bd5ff772152b9410" is_pre_election: true
I20240501 14:05:23.654855 5190 leader_election.cc:302] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, e9c60449cf3c46f8aca4ab6c535302a4; no voters:
I20240501 14:05:23.655068 4992 raft_consensus.cc:2454] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 0.
I20240501 14:05:23.655480 6711 raft_consensus.cc:2781] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240501 14:05:23.655649 6711 raft_consensus.cc:482] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240501 14:05:23.655753 6711 raft_consensus.cc:3037] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:23.658071 6711 raft_consensus.cc:504] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } }
I20240501 14:05:23.658598 6711 leader_election.cc:288] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 1 election: Requested vote from peers 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129:44033), e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377)
I20240501 14:05:23.659076 5126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec1e81011c6a46bc8a39aecf3a5e71e4" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9c60449cf3c46f8aca4ab6c535302a4"
I20240501 14:05:23.659091 4992 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec1e81011c6a46bc8a39aecf3a5e71e4" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "04088ee3bea440a7bd5ff772152b9410"
I20240501 14:05:23.659277 4992 raft_consensus.cc:3037] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 04088ee3bea440a7bd5ff772152b9410 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:23.659294 5126 raft_consensus.cc:3037] T ec1e81011c6a46bc8a39aecf3a5e71e4 P e9c60449cf3c46f8aca4ab6c535302a4 [term 0 FOLLOWER]: Advancing to term 1
I20240501 14:05:23.661885 4992 raft_consensus.cc:2454] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 1.
I20240501 14:05:23.661947 5126 raft_consensus.cc:2454] T ec1e81011c6a46bc8a39aecf3a5e71e4 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 1.
I20240501 14:05:23.662497 5190 leader_election.cc:302] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, e9c60449cf3c46f8aca4ab6c535302a4; no voters:
I20240501 14:05:23.662758 6711 raft_consensus.cc:2781] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Leader election won for term 1
I20240501 14:05:23.663326 6711 raft_consensus.cc:686] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEADER]: Becoming Leader. State: Replica: 1c3fdc5272484ac09ef3e5e36a08f27c, State: Running, Role: LEADER
I20240501 14:05:23.663594 6711 consensus_queue.cc:237] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } }
I20240501 14:05:23.667169 4849 catalog_manager.cc:5484] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c reported cstate change: term changed from 0 to 1, leader changed from <none> to 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131). New cstate: current_term: 1 leader_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: UNKNOWN } } }
I20240501 14:05:23.790297 6750 consensus_queue.cc:1035] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:05:23.794672 6751 consensus_queue.cc:1035] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:05:23.986116 6751 consensus_queue.cc:1035] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240501 14:05:23.992092 6750 consensus_queue.cc:1035] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:05:24.092063 6756 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:05:24.092339 6756 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240501 14:05:24.097295 6711 consensus_queue.cc:1035] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240501 14:05:24.100718 6756 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:05:24.101608 6768 consensus_queue.cc:1035] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_9'
W20240501 14:05:24.326932 5190 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.124.130:34377, user_credentials={real_user=slave}} blocked reactor thread for 61681.3us
INSERT report
rows total: 2000
time total: 201.661 ms
time per row: 0.100831 ms
I20240501 14:05:24.453244 30194 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUv7aek/build/asan/bin/kudu
/tmp/dist-test-taskUv7aek/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.124.133:35635
--local_ip_for_outbound_sockets=127.29.124.133
--tserver_master_addrs=127.29.124.190:35761
--webserver_port=46525
--webserver_interface=127.29.124.133
--builtin_ntp_servers=127.29.124.148:46431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240501 14:05:24.799348 6810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:05:24.799669 6810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:05:24.799803 6810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:05:24.808764 6810 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:05:24.808876 6810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:05:24.808997 6810 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.124.133
I20240501 14:05:24.818267 6810 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.124.148:46431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.124.133:35635
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.29.124.133
--webserver_port=46525
--tserver_master_addrs=127.29.124.190:35761
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.124.133
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:05:24.819553 6810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:05:24.821358 6810 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:05:24.833204 6815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:05:24.833564 6818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:05:24.834043 6816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:05:24.834323 6810 server_base.cc:1034] running on GCE node
I20240501 14:05:24.834851 6810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:05:24.835716 6810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:05:24.836910 6810 hybrid_clock.cc:648] HybridClock initialized: now 1714572324836863 us; error 37 us; skew 500 ppm
I20240501 14:05:24.837232 6810 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:05:24.839543 6810 webserver.cc:460] Webserver started at http://127.29.124.133:46525/ using document root <none> and password file <none>
I20240501 14:05:24.840260 6810 fs_manager.cc:359] Metadata directory not provided
I20240501 14:05:24.840384 6810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:05:24.845623 6810 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20240501 14:05:24.848348 6824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:05:24.849790 6810 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:05:24.849939 6810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "938a24b0e6a040b7966225af26558ff8"
format_stamp: "Formatted at 2024-05-01 14:05:06 on dist-test-slave-p83k"
I20240501 14:05:24.850720 6810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:05:24.868877 6810 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:05:24.869728 6810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:05:24.869997 6810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:05:24.870841 6810 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:05:24.872267 6810 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:05:24.872341 6810 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:05:24.872460 6810 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:05:24.872511 6810 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:05:24.902055 6810 rpc_server.cc:307] RPC server started. Bound to: 127.29.124.133:35635
I20240501 14:05:24.902105 6936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.124.133:35635 every 8 connection(s)
I20240501 14:05:24.903466 6810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240501 14:05:24.910248 30194 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUv7aek/build/asan/bin/kudu as pid 6810
I20240501 14:05:24.910879 30194 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUv7aek/build/asan/bin/kudu
/tmp/dist-test-taskUv7aek/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.124.134:40665
--local_ip_for_outbound_sockets=127.29.124.134
--tserver_master_addrs=127.29.124.190:35761
--webserver_port=37279
--webserver_interface=127.29.124.134
--builtin_ntp_servers=127.29.124.148:46431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:05:24.915953 6937 heartbeater.cc:347] Connected to a master server at 127.29.124.190:35761
I20240501 14:05:24.916218 6937 heartbeater.cc:464] Registering TS with master...
I20240501 14:05:24.917026 6937 heartbeater.cc:511] Master 127.29.124.190:35761 requested a full tablet report, sending...
I20240501 14:05:24.918713 4849 ts_manager.cc:194] Re-registered known tserver with Master: 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:05:24.919662 4849 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.124.133:33203
W20240501 14:05:25.255995 6941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:05:25.256270 6941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:05:25.256397 6941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:05:25.263856 6941 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:05:25.263988 6941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:05:25.264132 6941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.124.134
I20240501 14:05:25.273478 6941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.124.148:46431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.124.134:40665
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.29.124.134
--webserver_port=37279
--tserver_master_addrs=127.29.124.190:35761
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.124.134
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:05:25.274766 6941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:05:25.276487 6941 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:05:25.288120 6948 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:05:25.288671 6951 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:05:25.289181 6949 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:05:25.289496 6941 server_base.cc:1034] running on GCE node
I20240501 14:05:25.290032 6941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:05:25.290961 6941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:05:25.292158 6941 hybrid_clock.cc:648] HybridClock initialized: now 1714572325292119 us; error 33 us; skew 500 ppm
I20240501 14:05:25.292474 6941 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:05:25.294641 6941 webserver.cc:460] Webserver started at http://127.29.124.134:37279/ using document root <none> and password file <none>
I20240501 14:05:25.295341 6941 fs_manager.cc:359] Metadata directory not provided
I20240501 14:05:25.295480 6941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:05:25.300623 6941 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20240501 14:05:25.303290 6957 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:05:25.304613 6941 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 14:05:25.304760 6941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "19a6d2d8a08142d5bffcde3a37234692"
format_stamp: "Formatted at 2024-05-01 14:05:07 on dist-test-slave-p83k"
I20240501 14:05:25.305584 6941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:05:25.322338 6941 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:05:25.323082 6941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:05:25.323338 6941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:05:25.324159 6941 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:05:25.325628 6941 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:05:25.325704 6941 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:05:25.325825 6941 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:05:25.325876 6941 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:05:25.354393 6941 rpc_server.cc:307] RPC server started. Bound to: 127.29.124.134:40665
I20240501 14:05:25.354442 7069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.124.134:40665 every 8 connection(s)
I20240501 14:05:25.355856 6941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240501 14:05:25.359877 30194 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUv7aek/build/asan/bin/kudu as pid 6941
I20240501 14:05:25.360582 30194 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUv7aek/build/asan/bin/kudu
/tmp/dist-test-taskUv7aek/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.124.135:35329
--local_ip_for_outbound_sockets=127.29.124.135
--tserver_master_addrs=127.29.124.190:35761
--webserver_port=37713
--webserver_interface=127.29.124.135
--builtin_ntp_servers=127.29.124.148:46431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:05:25.369237 7070 heartbeater.cc:347] Connected to a master server at 127.29.124.190:35761
I20240501 14:05:25.369554 7070 heartbeater.cc:464] Registering TS with master...
I20240501 14:05:25.370476 7070 heartbeater.cc:511] Master 127.29.124.190:35761 requested a full tablet report, sending...
I20240501 14:05:25.371866 4849 ts_manager.cc:194] Re-registered known tserver with Master: 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665)
I20240501 14:05:25.372637 4849 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.124.134:53769
W20240501 14:05:25.696570 7073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:05:25.696841 7073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:05:25.696959 7073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:05:25.704948 7073 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:05:25.705097 7073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:05:25.705325 7073 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.124.135
I20240501 14:05:25.714596 7073 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.124.148:46431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.124.135:35329
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.29.124.135
--webserver_port=37713
--tserver_master_addrs=127.29.124.190:35761
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.124.135
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:05:25.715824 7073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:05:25.717481 7073 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:05:25.728514 7079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:05:25.729076 7080 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:05:25.729540 7082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:05:25.729770 7073 server_base.cc:1034] running on GCE node
I20240501 14:05:25.730352 7073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:05:25.731456 7073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:05:25.732700 7073 hybrid_clock.cc:648] HybridClock initialized: now 1714572325732641 us; error 50 us; skew 500 ppm
I20240501 14:05:25.732992 7073 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:05:25.735275 7073 webserver.cc:460] Webserver started at http://127.29.124.135:37713/ using document root <none> and password file <none>
I20240501 14:05:25.735958 7073 fs_manager.cc:359] Metadata directory not provided
I20240501 14:05:25.736076 7073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:05:25.741061 7073 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20240501 14:05:25.743628 7088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:05:25.744870 7073 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240501 14:05:25.745033 7073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "4360b59e3a154688a79ca5f9c0a37bc9"
format_stamp: "Formatted at 2024-05-01 14:05:07 on dist-test-slave-p83k"
I20240501 14:05:25.746028 7073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:05:25.776783 7073 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:05:25.777762 7073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:05:25.778061 7073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:05:25.779058 7073 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:05:25.780520 7073 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:05:25.780594 7073 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:05:25.780725 7073 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:05:25.780776 7073 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:05:25.808290 7073 rpc_server.cc:307] RPC server started. Bound to: 127.29.124.135:35329
I20240501 14:05:25.808352 7200 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.124.135:35329 every 8 connection(s)
I20240501 14:05:25.809703 7073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240501 14:05:25.811451 30194 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUv7aek/build/asan/bin/kudu as pid 7073
I20240501 14:05:25.812104 30194 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskUv7aek/build/asan/bin/kudu
/tmp/dist-test-taskUv7aek/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.29.124.136:43377
--local_ip_for_outbound_sockets=127.29.124.136
--tserver_master_addrs=127.29.124.190:35761
--webserver_port=37805
--webserver_interface=127.29.124.136
--builtin_ntp_servers=127.29.124.148:46431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240501 14:05:25.822947 7201 heartbeater.cc:347] Connected to a master server at 127.29.124.190:35761
I20240501 14:05:25.823251 7201 heartbeater.cc:464] Registering TS with master...
I20240501 14:05:25.823967 7201 heartbeater.cc:511] Master 127.29.124.190:35761 requested a full tablet report, sending...
I20240501 14:05:25.825306 4849 ts_manager.cc:194] Re-registered known tserver with Master: 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329)
I20240501 14:05:25.826047 4849 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.124.135:54407
I20240501 14:05:25.922045 6937 heartbeater.cc:503] Master 127.29.124.190:35761 was elected leader, sending a full tablet report...
W20240501 14:05:26.156260 7204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:05:26.156528 7204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240501 14:05:26.156653 7204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:05:26.164526 7204 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240501 14:05:26.164690 7204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240501 14:05:26.164875 7204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.29.124.136
I20240501 14:05:26.177886 7204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.29.124.148:46431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.29.124.136:43377
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.29.124.136
--webserver_port=37805
--tserver_master_addrs=127.29.124.190:35761
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.29.124.136
--log_dir=/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision 0d95304fa46ee5d96bcaa934c7660369f2860e06
build type FASTDEBUG
built by None at 01 May 2024 13:43:20 UTC on efd1337a2aa3
build id 75
ASAN enabled
I20240501 14:05:26.179165 7204 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240501 14:05:26.180811 7204 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240501 14:05:26.192816 7211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:05:26.193197 7213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240501 14:05:26.193619 7210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240501 14:05:26.193835 7204 server_base.cc:1034] running on GCE node
I20240501 14:05:26.194589 7204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240501 14:05:26.195497 7204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240501 14:05:26.196758 7204 hybrid_clock.cc:648] HybridClock initialized: now 1714572326196710 us; error 42 us; skew 500 ppm
I20240501 14:05:26.197136 7204 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240501 14:05:26.199299 7204 webserver.cc:460] Webserver started at http://127.29.124.136:37805/ using document root <none> and password file <none>
I20240501 14:05:26.200031 7204 fs_manager.cc:359] Metadata directory not provided
I20240501 14:05:26.200145 7204 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240501 14:05:26.205801 7204 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20240501 14:05:26.208456 7219 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240501 14:05:26.209970 7204 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240501 14:05:26.210136 7204 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "6dac90f01e744f9c8701a404440a4c77"
format_stamp: "Formatted at 2024-05-01 14:05:08 on dist-test-slave-p83k"
I20240501 14:05:26.210922 7204 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240501 14:05:26.231547 7204 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240501 14:05:26.232359 7204 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240501 14:05:26.232622 7204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240501 14:05:26.233547 7204 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240501 14:05:26.235003 7204 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240501 14:05:26.235085 7204 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240501 14:05:26.235199 7204 ts_tablet_manager.cc:590] Registered 0 tablets
I20240501 14:05:26.235249 7204 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240501 14:05:26.266896 7204 rpc_server.cc:307] RPC server started. Bound to: 127.29.124.136:43377
I20240501 14:05:26.266939 7332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.29.124.136:43377 every 8 connection(s)
I20240501 14:05:26.268317 7204 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240501 14:05:26.271661 30194 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskUv7aek/build/asan/bin/kudu as pid 7204
I20240501 14:05:26.288363 7333 heartbeater.cc:347] Connected to a master server at 127.29.124.190:35761
I20240501 14:05:26.288729 7333 heartbeater.cc:464] Registering TS with master...
I20240501 14:05:26.289783 7333 heartbeater.cc:511] Master 127.29.124.190:35761 requested a full tablet report, sending...
I20240501 14:05:26.291399 4849 ts_manager.cc:194] Re-registered known tserver with Master: 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377)
I20240501 14:05:26.292604 4849 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.29.124.136:33029
I20240501 14:05:26.375171 7070 heartbeater.cc:503] Master 127.29.124.190:35761 was elected leader, sending a full tablet report...
I20240501 14:05:26.828301 7201 heartbeater.cc:503] Master 127.29.124.190:35761 was elected leader, sending a full tablet report...
I20240501 14:05:27.295099 7333 heartbeater.cc:503] Master 127.29.124.190:35761 was elected leader, sending a full tablet report...
I20240501 14:05:27.467162 5234 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:27.471400 4973 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:27.472347 4973 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:27.565744 7136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:27.567140 5234 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:27.571300 6872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:27.585139 7267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:27.598161 7136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:27.649741 5234 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:27.670886 7267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:27.681349 4973 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:27.704571 7005 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:27.705216 7005 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:27.707306 7004 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:27.722678 5106 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:27.726472 5106 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:27.742245 7267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:27.750224 7136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:27.760885 5365 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:27.774410 6872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:27.831092 5106 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:27.831880 5365 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:27.872476 6872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:27.882292 5365 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:28.456583 7004 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:28.462818 4973 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:28.479027 5234 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:28.510664 7136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:28.535198 7004 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:28.547570 4973 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:28.571842 7136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:28.611719 5106 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:28.614419 5234 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:28.637449 5365 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:28.646880 7267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:28.656718 6872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:28.681303 7267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:28.693203 5106 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:28.731603 5365 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:28.750468 6872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:28.916899 4973 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:28.953229 5234 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:28.977150 7004 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:28.996146 7004 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:29.000932 4973 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:29.004647 7136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:29.037225 7267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:29.042753 5234 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:29.051584 7136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:29.056576 6872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:29.070312 5106 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:29.095338 5365 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:29.128137 7267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:29.155934 6872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:29.177000 5106 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:29.208120 5365 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:29.358739 7004 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:29.381029 4973 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:29.391733 5234 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:29.422351 7136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:29.460649 6872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:29.494266 5365 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:29.502616 5106 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:29.510035 7267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:30.081332 4973 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:30.117127 7004 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:30.150589 7136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:30.185047 7267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:30.197793 4973 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:30.209755 4973 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:30.245976 6872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:30.267051 5234 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:30.267529 7004 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:30.283304 5234 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:30.283356 5236 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:30.294993 6872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:30.331991 6872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:30.353076 5106 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:30.366007 7267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:30.379984 7136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:30.385524 7136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:30.453122 7267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:30.460312 7004 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:30.470330 5106 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:30.483168 5365 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:30.522444 5106 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:30.537597 5365 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:30.581350 5365 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:30.610956 4973 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:30.619935 7004 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:30.639045 7136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:30.656774 7267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:30.701535 5236 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:30.710536 6872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:30.737066 5365 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:30.760398 5106 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:31.006857 4973 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:31.020511 7004 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:31.038038 5256 consensus_queue.cc:237] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } }
I20240501 14:05:31.050618 7136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:31.061507 5236 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:05:31.094331 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: de6c7753c8564107b858876fcfa5924d. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:31.097751 4991 raft_consensus.cc:1261] T de6c7753c8564107b858876fcfa5924d P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:05:31.098452 7954 consensus_queue.cc:1035] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:05:31.099417 7267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:31.121076 5385 raft_consensus.cc:1261] T de6c7753c8564107b858876fcfa5924d P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:05:31.121883 7954 consensus_queue.cc:1035] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:05:31.133520 6872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:31.149065 7954 raft_consensus.cc:2932] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:31.155215 5384 raft_consensus.cc:2932] T de6c7753c8564107b858876fcfa5924d P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:31.159449 4851 catalog_manager.cc:5484] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c reported cstate change: config changed from index -1 to 4, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New cstate: current_term: 1 leader_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:05:31.165761 4995 raft_consensus.cc:2932] T de6c7753c8564107b858876fcfa5924d P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:31.196082 5365 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
W20240501 14:05:31.241545 5190 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.124.129:44033, user_credentials={real_user=slave}} blocked reactor thread for 69486.6us
I20240501 14:05:31.247604 5106 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:31.301904 4991 consensus_queue.cc:237] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } }
I20240501 14:05:31.324189 5385 raft_consensus.cc:1261] T fee16905387d428aa02cd2070c86c790 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:05:31.324894 7288 consensus_queue.cc:1035] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:05:31.334935 5125 raft_consensus.cc:1261] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:05:31.337512 7288 consensus_queue.cc:1035] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:05:31.399915 4930 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fee16905387d428aa02cd2070c86c790. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:31.403255 8043 raft_consensus.cc:2932] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:31.434055 4847 catalog_manager.cc:5484] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 reported cstate change: config changed from index -1 to 4, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) added. New cstate: current_term: 1 leader_uuid: "04088ee3bea440a7bd5ff772152b9410" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:05:31.445655 5126 raft_consensus.cc:2932] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:31.446161 5385 raft_consensus.cc:2932] T fee16905387d428aa02cd2070c86c790 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:31.518738 5256 consensus_queue.cc:237] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } }
I20240501 14:05:31.522938 8079 ts_tablet_manager.cc:902] T de6c7753c8564107b858876fcfa5924d P 6dac90f01e744f9c8701a404440a4c77: Initiating tablet copy from peer 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209)
I20240501 14:05:31.528558 5126 raft_consensus.cc:1261] T 1a7bbfc15a76450bbc9197037e856e08 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:05:31.528602 5385 raft_consensus.cc:1261] T 1a7bbfc15a76450bbc9197037e856e08 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:05:31.529232 8014 consensus_queue.cc:1035] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:05:31.534473 8014 consensus_queue.cc:1035] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:05:31.556208 8014 raft_consensus.cc:2932] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } }
I20240501 14:05:31.557988 5125 raft_consensus.cc:2932] T 1a7bbfc15a76450bbc9197037e856e08 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } }
I20240501 14:05:31.563943 4851 catalog_manager.cc:5484] T 1a7bbfc15a76450bbc9197037e856e08 P e9c60449cf3c46f8aca4ab6c535302a4 reported cstate change: config changed from index -1 to 4, NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) added. New cstate: current_term: 1 leader_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } }
I20240501 14:05:31.581732 5384 raft_consensus.cc:2932] T 1a7bbfc15a76450bbc9197037e856e08 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } }
W20240501 14:05:31.600019 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1a7bbfc15a76450bbc9197037e856e08. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:31.625510 5385 consensus_queue.cc:237] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }
I20240501 14:05:31.638927 8079 tablet_copy_client.cc:308] T de6c7753c8564107b858876fcfa5924d P 6dac90f01e744f9c8701a404440a4c77: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.131:33209
I20240501 14:05:31.640766 5256 raft_consensus.cc:1261] T 2c39b53867e047faaf24cc6f87a8100c P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:05:31.641876 8018 consensus_queue.cc:1035] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:05:31.651803 5126 raft_consensus.cc:1261] T 2c39b53867e047faaf24cc6f87a8100c P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:05:31.652401 8018 consensus_queue.cc:1035] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:05:31.664258 8018 raft_consensus.cc:2932] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:31.666546 5255 raft_consensus.cc:2932] T 2c39b53867e047faaf24cc6f87a8100c P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:31.669636 5276 tablet_copy_service.cc:140] P 1c3fdc5272484ac09ef3e5e36a08f27c: Received BeginTabletCopySession request for tablet de6c7753c8564107b858876fcfa5924d from peer 6dac90f01e744f9c8701a404440a4c77 ({username='slave'} at 127.29.124.136:49141)
I20240501 14:05:31.669967 5276 tablet_copy_service.cc:161] P 1c3fdc5272484ac09ef3e5e36a08f27c: Beginning new tablet copy session on tablet de6c7753c8564107b858876fcfa5924d from peer 6dac90f01e744f9c8701a404440a4c77 at {username='slave'} at 127.29.124.136:49141: session id = 6dac90f01e744f9c8701a404440a4c77-de6c7753c8564107b858876fcfa5924d
I20240501 14:05:31.673986 5126 raft_consensus.cc:2932] T 2c39b53867e047faaf24cc6f87a8100c P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:31.677076 5276 tablet_copy_source_session.cc:205] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:05:31.674197 4851 catalog_manager.cc:5484] T 2c39b53867e047faaf24cc6f87a8100c P 1c3fdc5272484ac09ef3e5e36a08f27c reported cstate change: config changed from index -1 to 4, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:31.690986 8079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de6c7753c8564107b858876fcfa5924d. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 14:05:31.710000 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2c39b53867e047faaf24cc6f87a8100c. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:31.769533 8139 ts_tablet_manager.cc:902] T fee16905387d428aa02cd2070c86c790 P 4360b59e3a154688a79ca5f9c0a37bc9: Initiating tablet copy from peer 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129:44033)
I20240501 14:05:31.777245 5126 consensus_queue.cc:237] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } }
I20240501 14:05:31.782548 8079 tablet_copy_client.cc:783] T de6c7753c8564107b858876fcfa5924d P 6dac90f01e744f9c8701a404440a4c77: tablet copy: Starting download of 0 data blocks...
I20240501 14:05:31.782922 8079 tablet_copy_client.cc:647] T de6c7753c8564107b858876fcfa5924d P 6dac90f01e744f9c8701a404440a4c77: tablet copy: Starting download of 1 WAL segments...
I20240501 14:05:31.785813 5125 consensus_queue.cc:237] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } }
I20240501 14:05:31.789479 5385 raft_consensus.cc:1261] T ac4b234f794149e4b6def7a3b358942d P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:05:31.791280 7843 consensus_queue.cc:1035] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:05:31.807986 8079 tablet_copy_client.cc:523] T de6c7753c8564107b858876fcfa5924d P 6dac90f01e744f9c8701a404440a4c77: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:05:31.814114 4991 raft_consensus.cc:1261] T 9d32e0707d244f2c9bbcb57f62b09f31 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:05:31.814790 7843 consensus_queue.cc:1035] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:05:31.824568 8079 tablet_bootstrap.cc:492] T de6c7753c8564107b858876fcfa5924d P 6dac90f01e744f9c8701a404440a4c77: Bootstrap starting.
I20240501 14:05:31.831984 5384 raft_consensus.cc:1261] T 9d32e0707d244f2c9bbcb57f62b09f31 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:05:31.833045 8149 consensus_queue.cc:1035] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:05:31.834064 4995 raft_consensus.cc:1261] T ac4b234f794149e4b6def7a3b358942d P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:05:31.840548 7843 consensus_queue.cc:1035] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:05:31.842118 7843 raft_consensus.cc:2932] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:31.844194 5384 raft_consensus.cc:2932] T 9d32e0707d244f2c9bbcb57f62b09f31 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:31.855171 4853 catalog_manager.cc:5484] T 9d32e0707d244f2c9bbcb57f62b09f31 P be56773105e744919601127a94e98adf reported cstate change: config changed from index -1 to 4, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New cstate: current_term: 1 leader_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:31.884371 4991 raft_consensus.cc:2932] T 9d32e0707d244f2c9bbcb57f62b09f31 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:31.894680 8149 raft_consensus.cc:2932] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
W20240501 14:05:31.895272 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ac4b234f794149e4b6def7a3b358942d. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:31.896648 5385 raft_consensus.cc:2932] T ac4b234f794149e4b6def7a3b358942d P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:31.903676 4853 catalog_manager.cc:5484] T ac4b234f794149e4b6def7a3b358942d P be56773105e744919601127a94e98adf reported cstate change: config changed from index -1 to 4, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) added. New cstate: current_term: 1 leader_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:31.906533 7004 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:31.914781 4995 raft_consensus.cc:2932] T ac4b234f794149e4b6def7a3b358942d P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:31.926218 8079 log.cc:824] T de6c7753c8564107b858876fcfa5924d P 6dac90f01e744f9c8701a404440a4c77: Log is configured to *not* fsync() on all Append() calls
W20240501 14:05:31.930613 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9d32e0707d244f2c9bbcb57f62b09f31. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:31.935853 4973 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:31.943303 5236 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:31.951236 8139 tablet_copy_client.cc:308] T fee16905387d428aa02cd2070c86c790 P 4360b59e3a154688a79ca5f9c0a37bc9: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.129:44033
I20240501 14:05:31.953529 7136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:31.955857 4973 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:31.999500 7267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:32.009347 5236 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:32.020537 7004 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:32.054769 5015 tablet_copy_service.cc:140] P 04088ee3bea440a7bd5ff772152b9410: Received BeginTabletCopySession request for tablet fee16905387d428aa02cd2070c86c790 from peer 4360b59e3a154688a79ca5f9c0a37bc9 ({username='slave'} at 127.29.124.135:54069)
I20240501 14:05:32.055064 5015 tablet_copy_service.cc:161] P 04088ee3bea440a7bd5ff772152b9410: Beginning new tablet copy session on tablet fee16905387d428aa02cd2070c86c790 from peer 4360b59e3a154688a79ca5f9c0a37bc9 at {username='slave'} at 127.29.124.135:54069: session id = 4360b59e3a154688a79ca5f9c0a37bc9-fee16905387d428aa02cd2070c86c790
I20240501 14:05:32.061933 5015 tablet_copy_source_session.cc:205] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:05:32.062361 5365 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:32.085842 8139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fee16905387d428aa02cd2070c86c790. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:32.097090 5106 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:32.106056 4995 consensus_queue.cc:237] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }
I20240501 14:05:32.115780 5384 raft_consensus.cc:1261] T fee16905387d428aa02cd2070c86c790 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:32.116581 8045 consensus_queue.cc:1035] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:32.119305 8045 raft_consensus.cc:2932] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:32.123732 5384 raft_consensus.cc:2932] T fee16905387d428aa02cd2070c86c790 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:32.130172 4846 catalog_manager.cc:5484] T fee16905387d428aa02cd2070c86c790 P be56773105e744919601127a94e98adf reported cstate change: config changed from index 4 to 5, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New cstate: current_term: 1 leader_uuid: "04088ee3bea440a7bd5ff772152b9410" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:32.146133 4992 consensus_queue.cc:237] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } }
W20240501 14:05:32.152932 4930 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fee16905387d428aa02cd2070c86c790. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:32.163025 7267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:32.163792 5384 raft_consensus.cc:1261] T 8cc5eec4600640f7a293877c825f8d96 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:05:32.165645 8205 ts_tablet_manager.cc:902] T 1a7bbfc15a76450bbc9197037e856e08 P 19a6d2d8a08142d5bffcde3a37234692: Initiating tablet copy from peer 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209)
I20240501 14:05:32.169059 5125 raft_consensus.cc:1261] T 8cc5eec4600640f7a293877c825f8d96 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:05:32.170046 8214 consensus_queue.cc:1035] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:05:32.170789 7136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:32.178453 8214 consensus_queue.cc:1035] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:05:32.189891 5384 raft_consensus.cc:2932] T 8cc5eec4600640f7a293877c825f8d96 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } }
I20240501 14:05:32.205786 4846 catalog_manager.cc:5484] T 8cc5eec4600640f7a293877c825f8d96 P be56773105e744919601127a94e98adf reported cstate change: config changed from index -1 to 4, NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) added. New cstate: current_term: 1 leader_uuid: "04088ee3bea440a7bd5ff772152b9410" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } }
I20240501 14:05:32.208253 8222 raft_consensus.cc:2932] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } }
I20240501 14:05:32.216532 5126 raft_consensus.cc:1261] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:32.221427 8210 consensus_queue.cc:1035] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:32.222856 5126 raft_consensus.cc:2932] T 8cc5eec4600640f7a293877c825f8d96 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } }
I20240501 14:05:32.230932 5125 raft_consensus.cc:2932] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
W20240501 14:05:32.241725 4930 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fee16905387d428aa02cd2070c86c790. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:32.243549 6872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:32.259336 8139 tablet_copy_client.cc:783] T fee16905387d428aa02cd2070c86c790 P 4360b59e3a154688a79ca5f9c0a37bc9: tablet copy: Starting download of 0 data blocks...
I20240501 14:05:32.259716 8139 tablet_copy_client.cc:647] T fee16905387d428aa02cd2070c86c790 P 4360b59e3a154688a79ca5f9c0a37bc9: tablet copy: Starting download of 1 WAL segments...
I20240501 14:05:32.274101 8139 tablet_copy_client.cc:523] T fee16905387d428aa02cd2070c86c790 P 4360b59e3a154688a79ca5f9c0a37bc9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:05:32.278627 8139 tablet_bootstrap.cc:492] T fee16905387d428aa02cd2070c86c790 P 4360b59e3a154688a79ca5f9c0a37bc9: Bootstrap starting.
I20240501 14:05:32.294076 5365 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:32.296569 5106 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:32.297567 6872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:32.304805 8227 ts_tablet_manager.cc:902] T 2c39b53867e047faaf24cc6f87a8100c P 938a24b0e6a040b7966225af26558ff8: Initiating tablet copy from peer be56773105e744919601127a94e98adf (127.29.124.132:35459)
I20240501 14:05:32.320626 8139 log.cc:824] T fee16905387d428aa02cd2070c86c790 P 4360b59e3a154688a79ca5f9c0a37bc9: Log is configured to *not* fsync() on all Append() calls
I20240501 14:05:32.350934 8235 ts_tablet_manager.cc:902] T 9d32e0707d244f2c9bbcb57f62b09f31 P 6dac90f01e744f9c8701a404440a4c77: Initiating tablet copy from peer e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377)
W20240501 14:05:32.356076 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8cc5eec4600640f7a293877c825f8d96. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:32.385710 8235 tablet_copy_client.cc:308] T 9d32e0707d244f2c9bbcb57f62b09f31 P 6dac90f01e744f9c8701a404440a4c77: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.130:34377
I20240501 14:05:32.400626 8079 tablet_bootstrap.cc:492] T de6c7753c8564107b858876fcfa5924d P 6dac90f01e744f9c8701a404440a4c77: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:05:32.401497 8079 tablet_bootstrap.cc:492] T de6c7753c8564107b858876fcfa5924d P 6dac90f01e744f9c8701a404440a4c77: Bootstrap complete.
I20240501 14:05:32.402161 8079 ts_tablet_manager.cc:1367] T de6c7753c8564107b858876fcfa5924d P 6dac90f01e744f9c8701a404440a4c77: Time spent bootstrapping tablet: real 0.578s user 0.066s sys 0.003s
I20240501 14:05:32.416855 8079 raft_consensus.cc:348] T de6c7753c8564107b858876fcfa5924d P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } }
I20240501 14:05:32.417559 8079 raft_consensus.cc:729] T de6c7753c8564107b858876fcfa5924d P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6dac90f01e744f9c8701a404440a4c77, State: Initialized, Role: LEARNER
I20240501 14:05:32.418170 8079 consensus_queue.cc:260] T de6c7753c8564107b858876fcfa5924d P 6dac90f01e744f9c8701a404440a4c77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } }
I20240501 14:05:32.429936 8079 ts_tablet_manager.cc:1398] T de6c7753c8564107b858876fcfa5924d P 6dac90f01e744f9c8701a404440a4c77: Time spent starting tablet: real 0.027s user 0.012s sys 0.000s
I20240501 14:05:32.431751 8241 ts_tablet_manager.cc:902] T ac4b234f794149e4b6def7a3b358942d P 4360b59e3a154688a79ca5f9c0a37bc9: Initiating tablet copy from peer e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377)
W20240501 14:05:32.433548 5320 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.124.130:34377, user_credentials={real_user=slave}} blocked reactor thread for 48370.8us
I20240501 14:05:32.436453 5276 tablet_copy_service.cc:342] P 1c3fdc5272484ac09ef3e5e36a08f27c: Request end of tablet copy session 6dac90f01e744f9c8701a404440a4c77-de6c7753c8564107b858876fcfa5924d received from {username='slave'} at 127.29.124.136:49141
I20240501 14:05:32.436656 5276 tablet_copy_service.cc:433] P 1c3fdc5272484ac09ef3e5e36a08f27c: ending tablet copy session 6dac90f01e744f9c8701a404440a4c77-de6c7753c8564107b858876fcfa5924d on tablet de6c7753c8564107b858876fcfa5924d with peer 6dac90f01e744f9c8701a404440a4c77
I20240501 14:05:32.445027 5146 tablet_copy_service.cc:140] P e9c60449cf3c46f8aca4ab6c535302a4: Received BeginTabletCopySession request for tablet 9d32e0707d244f2c9bbcb57f62b09f31 from peer 6dac90f01e744f9c8701a404440a4c77 ({username='slave'} at 127.29.124.136:51401)
I20240501 14:05:32.445307 5146 tablet_copy_service.cc:161] P e9c60449cf3c46f8aca4ab6c535302a4: Beginning new tablet copy session on tablet 9d32e0707d244f2c9bbcb57f62b09f31 from peer 6dac90f01e744f9c8701a404440a4c77 at {username='slave'} at 127.29.124.136:51401: session id = 6dac90f01e744f9c8701a404440a4c77-9d32e0707d244f2c9bbcb57f62b09f31
I20240501 14:05:32.449530 5146 tablet_copy_source_session.cc:205] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:05:32.451418 8235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d32e0707d244f2c9bbcb57f62b09f31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:32.457592 5255 consensus_queue.cc:237] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } }
I20240501 14:05:32.461721 8235 tablet_copy_client.cc:783] T 9d32e0707d244f2c9bbcb57f62b09f31 P 6dac90f01e744f9c8701a404440a4c77: tablet copy: Starting download of 0 data blocks...
I20240501 14:05:32.461963 8235 tablet_copy_client.cc:647] T 9d32e0707d244f2c9bbcb57f62b09f31 P 6dac90f01e744f9c8701a404440a4c77: tablet copy: Starting download of 1 WAL segments...
I20240501 14:05:32.463816 8235 tablet_copy_client.cc:523] T 9d32e0707d244f2c9bbcb57f62b09f31 P 6dac90f01e744f9c8701a404440a4c77: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:05:32.470259 8235 tablet_bootstrap.cc:492] T 9d32e0707d244f2c9bbcb57f62b09f31 P 6dac90f01e744f9c8701a404440a4c77: Bootstrap starting.
I20240501 14:05:32.470742 5126 consensus_queue.cc:237] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } }
I20240501 14:05:32.497140 8241 tablet_copy_client.cc:308] T ac4b234f794149e4b6def7a3b358942d P 4360b59e3a154688a79ca5f9c0a37bc9: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.130:34377
I20240501 14:05:32.506261 5146 tablet_copy_service.cc:140] P e9c60449cf3c46f8aca4ab6c535302a4: Received BeginTabletCopySession request for tablet ac4b234f794149e4b6def7a3b358942d from peer 4360b59e3a154688a79ca5f9c0a37bc9 ({username='slave'} at 127.29.124.135:58119)
I20240501 14:05:32.506458 5146 tablet_copy_service.cc:161] P e9c60449cf3c46f8aca4ab6c535302a4: Beginning new tablet copy session on tablet ac4b234f794149e4b6def7a3b358942d from peer 4360b59e3a154688a79ca5f9c0a37bc9 at {username='slave'} at 127.29.124.135:58119: session id = 4360b59e3a154688a79ca5f9c0a37bc9-ac4b234f794149e4b6def7a3b358942d
I20240501 14:05:32.508884 4992 raft_consensus.cc:1261] T 9fd94635797d447cbdc9181ee91f465b P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:05:32.509431 5146 tablet_copy_source_session.cc:205] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:05:32.509546 7689 consensus_queue.cc:1035] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:05:32.510715 8241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac4b234f794149e4b6def7a3b358942d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:32.513763 5384 raft_consensus.cc:1261] T f9cb784f60774ea7b6e70ccfebf867a8 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:05:32.513813 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f9cb784f60774ea7b6e70ccfebf867a8. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:32.514182 5254 raft_consensus.cc:1261] T f9cb784f60774ea7b6e70ccfebf867a8 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:05:32.514642 8149 consensus_queue.cc:1035] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:05:32.515228 8175 consensus_queue.cc:1035] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:05:32.519225 8241 tablet_copy_client.cc:783] T ac4b234f794149e4b6def7a3b358942d P 4360b59e3a154688a79ca5f9c0a37bc9: tablet copy: Starting download of 0 data blocks...
I20240501 14:05:32.519513 8241 tablet_copy_client.cc:647] T ac4b234f794149e4b6def7a3b358942d P 4360b59e3a154688a79ca5f9c0a37bc9: tablet copy: Starting download of 1 WAL segments...
I20240501 14:05:32.521148 8241 tablet_copy_client.cc:523] T ac4b234f794149e4b6def7a3b358942d P 4360b59e3a154688a79ca5f9c0a37bc9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:05:32.527544 5126 raft_consensus.cc:1261] T 9fd94635797d447cbdc9181ee91f465b P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:05:32.528470 7689 consensus_queue.cc:1035] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:05:32.530030 8241 tablet_bootstrap.cc:492] T ac4b234f794149e4b6def7a3b358942d P 4360b59e3a154688a79ca5f9c0a37bc9: Bootstrap starting.
I20240501 14:05:32.538394 8139 tablet_bootstrap.cc:492] T fee16905387d428aa02cd2070c86c790 P 4360b59e3a154688a79ca5f9c0a37bc9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:05:32.539765 7004 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:32.546190 5254 raft_consensus.cc:2932] T f9cb784f60774ea7b6e70ccfebf867a8 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:32.548422 8139 tablet_bootstrap.cc:492] T fee16905387d428aa02cd2070c86c790 P 4360b59e3a154688a79ca5f9c0a37bc9: Bootstrap complete.
I20240501 14:05:32.548905 8139 ts_tablet_manager.cc:1367] T fee16905387d428aa02cd2070c86c790 P 4360b59e3a154688a79ca5f9c0a37bc9: Time spent bootstrapping tablet: real 0.271s user 0.064s sys 0.009s
I20240501 14:05:32.553458 8227 tablet_copy_client.cc:308] T 2c39b53867e047faaf24cc6f87a8100c P 938a24b0e6a040b7966225af26558ff8: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.132:35459
I20240501 14:05:32.565093 8149 raft_consensus.cc:2932] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:32.566362 5406 tablet_copy_service.cc:140] P be56773105e744919601127a94e98adf: Received BeginTabletCopySession request for tablet 2c39b53867e047faaf24cc6f87a8100c from peer 938a24b0e6a040b7966225af26558ff8 ({username='slave'} at 127.29.124.133:50583)
I20240501 14:05:32.565433 8139 raft_consensus.cc:348] T fee16905387d428aa02cd2070c86c790 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } }
I20240501 14:05:32.566622 5406 tablet_copy_service.cc:161] P be56773105e744919601127a94e98adf: Beginning new tablet copy session on tablet 2c39b53867e047faaf24cc6f87a8100c from peer 938a24b0e6a040b7966225af26558ff8 at {username='slave'} at 127.29.124.133:50583: session id = 938a24b0e6a040b7966225af26558ff8-2c39b53867e047faaf24cc6f87a8100c
I20240501 14:05:32.568656 8139 raft_consensus.cc:729] T fee16905387d428aa02cd2070c86c790 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4360b59e3a154688a79ca5f9c0a37bc9, State: Initialized, Role: LEARNER
I20240501 14:05:32.570137 7287 raft_consensus.cc:1203] T de6c7753c8564107b858876fcfa5924d P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:05:32.572278 5406 tablet_copy_source_session.cc:205] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:05:32.575716 4973 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:32.578727 8227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c39b53867e047faaf24cc6f87a8100c. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 14:05:32.595925 5060 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.124.132:35459, user_credentials={real_user=slave}} blocked reactor thread for 51422.6us
I20240501 14:05:32.598022 8139 consensus_queue.cc:260] T fee16905387d428aa02cd2070c86c790 P 4360b59e3a154688a79ca5f9c0a37bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } }
I20240501 14:05:32.600263 8277 raft_consensus.cc:2932] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:32.600816 4846 catalog_manager.cc:5484] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 reported cstate change: config changed from index -1 to 4, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New cstate: current_term: 1 leader_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:05:32.602689 4995 raft_consensus.cc:2932] T 9fd94635797d447cbdc9181ee91f465b P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:32.612654 4846 catalog_manager.cc:5484] T 9fd94635797d447cbdc9181ee91f465b P 04088ee3bea440a7bd5ff772152b9410 reported cstate change: config changed from index -1 to 4, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) added. New cstate: current_term: 1 leader_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:32.615837 7136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 14:05:32.622478 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9fd94635797d447cbdc9181ee91f465b. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:32.633467 7267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:32.636754 8139 ts_tablet_manager.cc:1398] T fee16905387d428aa02cd2070c86c790 P 4360b59e3a154688a79ca5f9c0a37bc9: Time spent starting tablet: real 0.088s user 0.009s sys 0.000s
I20240501 14:05:32.637900 5015 tablet_copy_service.cc:342] P 04088ee3bea440a7bd5ff772152b9410: Request end of tablet copy session 4360b59e3a154688a79ca5f9c0a37bc9-fee16905387d428aa02cd2070c86c790 received from {username='slave'} at 127.29.124.135:54069
I20240501 14:05:32.638095 5015 tablet_copy_service.cc:433] P 04088ee3bea440a7bd5ff772152b9410: ending tablet copy session 4360b59e3a154688a79ca5f9c0a37bc9-fee16905387d428aa02cd2070c86c790 on tablet fee16905387d428aa02cd2070c86c790 with peer 4360b59e3a154688a79ca5f9c0a37bc9
I20240501 14:05:32.643328 8139 tablet_replica.cc:330] T fee16905387d428aa02cd2070c86c790 P 4360b59e3a154688a79ca5f9c0a37bc9: stopping tablet replica
I20240501 14:05:32.643389 5384 raft_consensus.cc:2932] T f9cb784f60774ea7b6e70ccfebf867a8 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:32.644021 8139 raft_consensus.cc:2229] T fee16905387d428aa02cd2070c86c790 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 LEARNER]: Raft consensus shutting down.
I20240501 14:05:32.644384 8139 raft_consensus.cc:2258] T fee16905387d428aa02cd2070c86c790 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 LEARNER]: Raft consensus is shut down!
I20240501 14:05:32.645918 8139 ts_tablet_manager.cc:1875] T fee16905387d428aa02cd2070c86c790 P 4360b59e3a154688a79ca5f9c0a37bc9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:05:32.666534 8285 ts_tablet_manager.cc:902] T 8cc5eec4600640f7a293877c825f8d96 P 19a6d2d8a08142d5bffcde3a37234692: Initiating tablet copy from peer 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129:44033)
I20240501 14:05:32.670068 8205 tablet_copy_client.cc:308] T 1a7bbfc15a76450bbc9197037e856e08 P 19a6d2d8a08142d5bffcde3a37234692: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.131:33209
I20240501 14:05:32.685998 8139 ts_tablet_manager.cc:1888] T fee16905387d428aa02cd2070c86c790 P 4360b59e3a154688a79ca5f9c0a37bc9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:05:32.686182 8139 log.cc:1196] T fee16905387d428aa02cd2070c86c790 P 4360b59e3a154688a79ca5f9c0a37bc9: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/wal/wals/fee16905387d428aa02cd2070c86c790
I20240501 14:05:32.686724 8139 ts_tablet_manager.cc:902] T fee16905387d428aa02cd2070c86c790 P 4360b59e3a154688a79ca5f9c0a37bc9: Initiating tablet copy from peer 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129:44033)
I20240501 14:05:32.693719 5276 tablet_copy_service.cc:140] P 1c3fdc5272484ac09ef3e5e36a08f27c: Received BeginTabletCopySession request for tablet 1a7bbfc15a76450bbc9197037e856e08 from peer 19a6d2d8a08142d5bffcde3a37234692 ({username='slave'} at 127.29.124.134:43435)
I20240501 14:05:32.693928 5276 tablet_copy_service.cc:161] P 1c3fdc5272484ac09ef3e5e36a08f27c: Beginning new tablet copy session on tablet 1a7bbfc15a76450bbc9197037e856e08 from peer 19a6d2d8a08142d5bffcde3a37234692 at {username='slave'} at 127.29.124.134:43435: session id = 19a6d2d8a08142d5bffcde3a37234692-1a7bbfc15a76450bbc9197037e856e08
I20240501 14:05:32.695616 8290 ts_tablet_manager.cc:902] T fee16905387d428aa02cd2070c86c790 P 938a24b0e6a040b7966225af26558ff8: Initiating tablet copy from peer 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129:44033)
I20240501 14:05:32.696712 5276 tablet_copy_source_session.cc:205] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:05:32.699772 5236 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:32.704320 8205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a7bbfc15a76450bbc9197037e856e08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:32.706154 5125 raft_consensus.cc:2932] T 9fd94635797d447cbdc9181ee91f465b P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:32.720624 8235 tablet_bootstrap.cc:492] T 9d32e0707d244f2c9bbcb57f62b09f31 P 6dac90f01e744f9c8701a404440a4c77: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:05:32.721089 8235 tablet_bootstrap.cc:492] T 9d32e0707d244f2c9bbcb57f62b09f31 P 6dac90f01e744f9c8701a404440a4c77: Bootstrap complete.
I20240501 14:05:32.730927 8285 tablet_copy_client.cc:308] T 8cc5eec4600640f7a293877c825f8d96 P 19a6d2d8a08142d5bffcde3a37234692: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.129:44033
I20240501 14:05:32.736115 6872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:32.737629 8235 ts_tablet_manager.cc:1367] T 9d32e0707d244f2c9bbcb57f62b09f31 P 6dac90f01e744f9c8701a404440a4c77: Time spent bootstrapping tablet: real 0.267s user 0.039s sys 0.009s
I20240501 14:05:32.739457 5015 tablet_copy_service.cc:140] P 04088ee3bea440a7bd5ff772152b9410: Received BeginTabletCopySession request for tablet 8cc5eec4600640f7a293877c825f8d96 from peer 19a6d2d8a08142d5bffcde3a37234692 ({username='slave'} at 127.29.124.134:36629)
I20240501 14:05:32.739634 5015 tablet_copy_service.cc:161] P 04088ee3bea440a7bd5ff772152b9410: Beginning new tablet copy session on tablet 8cc5eec4600640f7a293877c825f8d96 from peer 19a6d2d8a08142d5bffcde3a37234692 at {username='slave'} at 127.29.124.134:36629: session id = 19a6d2d8a08142d5bffcde3a37234692-8cc5eec4600640f7a293877c825f8d96
I20240501 14:05:32.742856 5015 tablet_copy_source_session.cc:205] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:05:32.744329 8285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cc5eec4600640f7a293877c825f8d96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:32.744509 8235 raft_consensus.cc:348] T 9d32e0707d244f2c9bbcb57f62b09f31 P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } }
I20240501 14:05:32.744782 8235 raft_consensus.cc:729] T 9d32e0707d244f2c9bbcb57f62b09f31 P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6dac90f01e744f9c8701a404440a4c77, State: Initialized, Role: LEARNER
I20240501 14:05:32.745004 8235 consensus_queue.cc:260] T 9d32e0707d244f2c9bbcb57f62b09f31 P 6dac90f01e744f9c8701a404440a4c77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } }
I20240501 14:05:32.748250 8235 ts_tablet_manager.cc:1398] T 9d32e0707d244f2c9bbcb57f62b09f31 P 6dac90f01e744f9c8701a404440a4c77: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240501 14:05:32.755796 8205 tablet_copy_client.cc:783] T 1a7bbfc15a76450bbc9197037e856e08 P 19a6d2d8a08142d5bffcde3a37234692: tablet copy: Starting download of 0 data blocks...
I20240501 14:05:32.756191 8205 tablet_copy_client.cc:647] T 1a7bbfc15a76450bbc9197037e856e08 P 19a6d2d8a08142d5bffcde3a37234692: tablet copy: Starting download of 1 WAL segments...
I20240501 14:05:32.763455 8205 tablet_copy_client.cc:523] T 1a7bbfc15a76450bbc9197037e856e08 P 19a6d2d8a08142d5bffcde3a37234692: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:05:32.766403 8227 tablet_copy_client.cc:783] T 2c39b53867e047faaf24cc6f87a8100c P 938a24b0e6a040b7966225af26558ff8: tablet copy: Starting download of 0 data blocks...
I20240501 14:05:32.767298 8227 tablet_copy_client.cc:647] T 2c39b53867e047faaf24cc6f87a8100c P 938a24b0e6a040b7966225af26558ff8: tablet copy: Starting download of 1 WAL segments...
I20240501 14:05:32.774243 8285 tablet_copy_client.cc:783] T 8cc5eec4600640f7a293877c825f8d96 P 19a6d2d8a08142d5bffcde3a37234692: tablet copy: Starting download of 0 data blocks...
I20240501 14:05:32.776079 8285 tablet_copy_client.cc:647] T 8cc5eec4600640f7a293877c825f8d96 P 19a6d2d8a08142d5bffcde3a37234692: tablet copy: Starting download of 1 WAL segments...
I20240501 14:05:32.778208 5365 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:32.777905 8285 tablet_copy_client.cc:523] T 8cc5eec4600640f7a293877c825f8d96 P 19a6d2d8a08142d5bffcde3a37234692: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:05:32.781991 8285 tablet_bootstrap.cc:492] T 8cc5eec4600640f7a293877c825f8d96 P 19a6d2d8a08142d5bffcde3a37234692: Bootstrap starting.
I20240501 14:05:32.785292 8205 tablet_bootstrap.cc:492] T 1a7bbfc15a76450bbc9197037e856e08 P 19a6d2d8a08142d5bffcde3a37234692: Bootstrap starting.
I20240501 14:05:32.789587 8227 tablet_copy_client.cc:523] T 2c39b53867e047faaf24cc6f87a8100c P 938a24b0e6a040b7966225af26558ff8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:05:32.798418 8241 tablet_bootstrap.cc:492] T ac4b234f794149e4b6def7a3b358942d P 4360b59e3a154688a79ca5f9c0a37bc9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:05:32.798804 8241 tablet_bootstrap.cc:492] T ac4b234f794149e4b6def7a3b358942d P 4360b59e3a154688a79ca5f9c0a37bc9: Bootstrap complete.
I20240501 14:05:32.799098 8241 ts_tablet_manager.cc:1367] T ac4b234f794149e4b6def7a3b358942d P 4360b59e3a154688a79ca5f9c0a37bc9: Time spent bootstrapping tablet: real 0.269s user 0.039s sys 0.014s
I20240501 14:05:32.800053 8241 raft_consensus.cc:348] T ac4b234f794149e4b6def7a3b358942d P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } }
I20240501 14:05:32.800307 8241 raft_consensus.cc:729] T ac4b234f794149e4b6def7a3b358942d P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4360b59e3a154688a79ca5f9c0a37bc9, State: Initialized, Role: LEARNER
I20240501 14:05:32.800513 8241 consensus_queue.cc:260] T ac4b234f794149e4b6def7a3b358942d P 4360b59e3a154688a79ca5f9c0a37bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } }
I20240501 14:05:32.802255 8227 tablet_bootstrap.cc:492] T 2c39b53867e047faaf24cc6f87a8100c P 938a24b0e6a040b7966225af26558ff8: Bootstrap starting.
I20240501 14:05:32.816251 8205 log.cc:824] T 1a7bbfc15a76450bbc9197037e856e08 P 19a6d2d8a08142d5bffcde3a37234692: Log is configured to *not* fsync() on all Append() calls
I20240501 14:05:32.826635 5146 tablet_copy_service.cc:342] P e9c60449cf3c46f8aca4ab6c535302a4: Request end of tablet copy session 6dac90f01e744f9c8701a404440a4c77-9d32e0707d244f2c9bbcb57f62b09f31 received from {username='slave'} at 127.29.124.136:51401
I20240501 14:05:32.826823 5146 tablet_copy_service.cc:433] P e9c60449cf3c46f8aca4ab6c535302a4: ending tablet copy session 6dac90f01e744f9c8701a404440a4c77-9d32e0707d244f2c9bbcb57f62b09f31 on tablet 9d32e0707d244f2c9bbcb57f62b09f31 with peer 6dac90f01e744f9c8701a404440a4c77
I20240501 14:05:32.827523 7287 raft_consensus.cc:1203] T 9d32e0707d244f2c9bbcb57f62b09f31 P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:05:32.833771 8290 tablet_copy_client.cc:308] T fee16905387d428aa02cd2070c86c790 P 938a24b0e6a040b7966225af26558ff8: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.129:44033
I20240501 14:05:32.841173 5015 tablet_copy_service.cc:140] P 04088ee3bea440a7bd5ff772152b9410: Received BeginTabletCopySession request for tablet fee16905387d428aa02cd2070c86c790 from peer 938a24b0e6a040b7966225af26558ff8 ({username='slave'} at 127.29.124.133:49901)
I20240501 14:05:32.841390 5015 tablet_copy_service.cc:161] P 04088ee3bea440a7bd5ff772152b9410: Beginning new tablet copy session on tablet fee16905387d428aa02cd2070c86c790 from peer 938a24b0e6a040b7966225af26558ff8 at {username='slave'} at 127.29.124.133:49901: session id = 938a24b0e6a040b7966225af26558ff8-fee16905387d428aa02cd2070c86c790
I20240501 14:05:32.844183 5015 tablet_copy_source_session.cc:205] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:05:32.845791 8290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fee16905387d428aa02cd2070c86c790. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:32.846225 5126 consensus_queue.cc:237] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } }
I20240501 14:05:32.852593 5384 consensus_queue.cc:237] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } }
I20240501 14:05:32.852967 8290 tablet_copy_client.cc:783] T fee16905387d428aa02cd2070c86c790 P 938a24b0e6a040b7966225af26558ff8: tablet copy: Starting download of 0 data blocks...
I20240501 14:05:32.853189 8290 tablet_copy_client.cc:647] T fee16905387d428aa02cd2070c86c790 P 938a24b0e6a040b7966225af26558ff8: tablet copy: Starting download of 1 WAL segments...
I20240501 14:05:32.857295 8139 tablet_copy_client.cc:308] T fee16905387d428aa02cd2070c86c790 P 4360b59e3a154688a79ca5f9c0a37bc9: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.129:44033
I20240501 14:05:32.857868 5015 tablet_copy_service.cc:140] P 04088ee3bea440a7bd5ff772152b9410: Received BeginTabletCopySession request for tablet fee16905387d428aa02cd2070c86c790 from peer 4360b59e3a154688a79ca5f9c0a37bc9 ({username='slave'} at 127.29.124.135:54069)
I20240501 14:05:32.858031 5015 tablet_copy_service.cc:161] P 04088ee3bea440a7bd5ff772152b9410: Beginning new tablet copy session on tablet fee16905387d428aa02cd2070c86c790 from peer 4360b59e3a154688a79ca5f9c0a37bc9 at {username='slave'} at 127.29.124.135:54069: session id = 4360b59e3a154688a79ca5f9c0a37bc9-fee16905387d428aa02cd2070c86c790
I20240501 14:05:32.860992 5015 tablet_copy_source_session.cc:205] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:05:32.861969 8139 ts_tablet_manager.cc:1875] T fee16905387d428aa02cd2070c86c790 P 4360b59e3a154688a79ca5f9c0a37bc9: Deleting tablet data with delete state TABLET_DATA_COPYING
W20240501 14:05:32.865556 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9afe3b20a43d4b5bb800af39f945948d. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:32.866750 4995 raft_consensus.cc:1261] T 9afe3b20a43d4b5bb800af39f945948d P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:05:32.867369 8279 consensus_queue.cc:1035] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:05:32.867580 5254 raft_consensus.cc:1261] T 9afe3b20a43d4b5bb800af39f945948d P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:05:32.868682 8279 consensus_queue.cc:1035] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:05:32.868882 5106 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:32.870839 8139 ts_tablet_manager.cc:1888] T fee16905387d428aa02cd2070c86c790 P 4360b59e3a154688a79ca5f9c0a37bc9: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.4
I20240501 14:05:32.871151 8139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fee16905387d428aa02cd2070c86c790. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:32.874393 8139 tablet_copy_client.cc:783] T fee16905387d428aa02cd2070c86c790 P 4360b59e3a154688a79ca5f9c0a37bc9: tablet copy: Starting download of 0 data blocks...
I20240501 14:05:32.874629 8139 tablet_copy_client.cc:647] T fee16905387d428aa02cd2070c86c790 P 4360b59e3a154688a79ca5f9c0a37bc9: tablet copy: Starting download of 1 WAL segments...
I20240501 14:05:32.882117 8279 raft_consensus.cc:2932] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:32.882524 4995 raft_consensus.cc:2932] T 9afe3b20a43d4b5bb800af39f945948d P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:32.884469 8290 tablet_copy_client.cc:523] T fee16905387d428aa02cd2070c86c790 P 938a24b0e6a040b7966225af26558ff8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:05:32.888569 4846 catalog_manager.cc:5484] T 9afe3b20a43d4b5bb800af39f945948d P 04088ee3bea440a7bd5ff772152b9410 reported cstate change: config changed from index -1 to 4, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New cstate: current_term: 1 leader_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:32.889729 8139 tablet_copy_client.cc:523] T fee16905387d428aa02cd2070c86c790 P 4360b59e3a154688a79ca5f9c0a37bc9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:05:32.894335 8139 tablet_bootstrap.cc:492] T fee16905387d428aa02cd2070c86c790 P 4360b59e3a154688a79ca5f9c0a37bc9: Bootstrap starting.
I20240501 14:05:32.904752 8241 ts_tablet_manager.cc:1398] T ac4b234f794149e4b6def7a3b358942d P 4360b59e3a154688a79ca5f9c0a37bc9: Time spent starting tablet: real 0.106s user 0.004s sys 0.000s
I20240501 14:05:32.905469 5146 tablet_copy_service.cc:342] P e9c60449cf3c46f8aca4ab6c535302a4: Request end of tablet copy session 4360b59e3a154688a79ca5f9c0a37bc9-ac4b234f794149e4b6def7a3b358942d received from {username='slave'} at 127.29.124.135:58119
I20240501 14:05:32.905596 5146 tablet_copy_service.cc:433] P e9c60449cf3c46f8aca4ab6c535302a4: ending tablet copy session 4360b59e3a154688a79ca5f9c0a37bc9-ac4b234f794149e4b6def7a3b358942d on tablet ac4b234f794149e4b6def7a3b358942d with peer 4360b59e3a154688a79ca5f9c0a37bc9
I20240501 14:05:32.911864 5126 raft_consensus.cc:1261] T f4d2b8c005b64bbb8e6621da068d15e5 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:05:32.912617 8247 consensus_queue.cc:1035] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:05:32.918426 8241 ts_tablet_manager.cc:902] T 9fd94635797d447cbdc9181ee91f465b P 4360b59e3a154688a79ca5f9c0a37bc9: Initiating tablet copy from peer 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209)
I20240501 14:05:32.919735 5254 raft_consensus.cc:2932] T 9afe3b20a43d4b5bb800af39f945948d P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:32.922086 5255 raft_consensus.cc:1261] T f4d2b8c005b64bbb8e6621da068d15e5 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:05:32.922837 8247 consensus_queue.cc:1035] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:05:32.944712 8247 raft_consensus.cc:2932] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } }
I20240501 14:05:32.948235 5125 raft_consensus.cc:2932] T f4d2b8c005b64bbb8e6621da068d15e5 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } }
I20240501 14:05:32.952888 4846 catalog_manager.cc:5484] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf reported cstate change: config changed from index -1 to 4, NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) added. New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:05:32.969635 8227 log.cc:824] T 2c39b53867e047faaf24cc6f87a8100c P 938a24b0e6a040b7966225af26558ff8: Log is configured to *not* fsync() on all Append() calls
I20240501 14:05:32.975528 8290 tablet_bootstrap.cc:492] T fee16905387d428aa02cd2070c86c790 P 938a24b0e6a040b7966225af26558ff8: Bootstrap starting.
W20240501 14:05:32.987924 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f4d2b8c005b64bbb8e6621da068d15e5. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:32.996399 5255 raft_consensus.cc:2932] T f4d2b8c005b64bbb8e6621da068d15e5 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } }
I20240501 14:05:33.005889 8241 tablet_copy_client.cc:308] T 9fd94635797d447cbdc9181ee91f465b P 4360b59e3a154688a79ca5f9c0a37bc9: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.131:33209
I20240501 14:05:33.024214 8235 ts_tablet_manager.cc:902] T f9cb784f60774ea7b6e70ccfebf867a8 P 6dac90f01e744f9c8701a404440a4c77: Initiating tablet copy from peer e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377)
I20240501 14:05:33.029091 5385 consensus_queue.cc:237] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }
I20240501 14:05:33.037513 4995 raft_consensus.cc:1261] T 9b88c43cb8704583a3e65c5c35c205f6 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:05:33.038403 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9b88c43cb8704583a3e65c5c35c205f6. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:33.038789 8118 consensus_queue.cc:1035] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:05:33.043532 5255 raft_consensus.cc:1261] T 9b88c43cb8704583a3e65c5c35c205f6 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:05:33.044056 8118 consensus_queue.cc:1035] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:05:33.058358 8139 tablet_bootstrap.cc:492] T fee16905387d428aa02cd2070c86c790 P 4360b59e3a154688a79ca5f9c0a37bc9: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:05:33.058782 8139 tablet_bootstrap.cc:492] T fee16905387d428aa02cd2070c86c790 P 4360b59e3a154688a79ca5f9c0a37bc9: Bootstrap complete.
I20240501 14:05:33.059057 8139 ts_tablet_manager.cc:1367] T fee16905387d428aa02cd2070c86c790 P 4360b59e3a154688a79ca5f9c0a37bc9: Time spent bootstrapping tablet: real 0.165s user 0.043s sys 0.021s
I20240501 14:05:33.059863 8139 raft_consensus.cc:348] T fee16905387d428aa02cd2070c86c790 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }
I20240501 14:05:33.060144 8139 raft_consensus.cc:729] T fee16905387d428aa02cd2070c86c790 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4360b59e3a154688a79ca5f9c0a37bc9, State: Initialized, Role: LEARNER
I20240501 14:05:33.063694 8235 tablet_copy_client.cc:308] T f9cb784f60774ea7b6e70ccfebf867a8 P 6dac90f01e744f9c8701a404440a4c77: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.130:34377
I20240501 14:05:33.060369 8139 consensus_queue.cc:260] T fee16905387d428aa02cd2070c86c790 P 4360b59e3a154688a79ca5f9c0a37bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }
I20240501 14:05:33.064401 5146 tablet_copy_service.cc:140] P e9c60449cf3c46f8aca4ab6c535302a4: Received BeginTabletCopySession request for tablet f9cb784f60774ea7b6e70ccfebf867a8 from peer 6dac90f01e744f9c8701a404440a4c77 ({username='slave'} at 127.29.124.136:51401)
I20240501 14:05:33.064575 5146 tablet_copy_service.cc:161] P e9c60449cf3c46f8aca4ab6c535302a4: Beginning new tablet copy session on tablet f9cb784f60774ea7b6e70ccfebf867a8 from peer 6dac90f01e744f9c8701a404440a4c77 at {username='slave'} at 127.29.124.136:51401: session id = 6dac90f01e744f9c8701a404440a4c77-f9cb784f60774ea7b6e70ccfebf867a8
I20240501 14:05:33.064733 8139 ts_tablet_manager.cc:1398] T fee16905387d428aa02cd2070c86c790 P 4360b59e3a154688a79ca5f9c0a37bc9: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20240501 14:05:33.065615 5015 tablet_copy_service.cc:342] P 04088ee3bea440a7bd5ff772152b9410: Request end of tablet copy session 4360b59e3a154688a79ca5f9c0a37bc9-fee16905387d428aa02cd2070c86c790 received from {username='slave'} at 127.29.124.135:54069
I20240501 14:05:33.065739 5015 tablet_copy_service.cc:433] P 04088ee3bea440a7bd5ff772152b9410: ending tablet copy session 4360b59e3a154688a79ca5f9c0a37bc9-fee16905387d428aa02cd2070c86c790 on tablet fee16905387d428aa02cd2070c86c790 with peer 4360b59e3a154688a79ca5f9c0a37bc9
I20240501 14:05:33.070335 8281 raft_consensus.cc:1050] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c: attempting to promote NON_VOTER 6dac90f01e744f9c8701a404440a4c77 to VOTER
I20240501 14:05:33.071362 8281 consensus_queue.cc:237] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } }
I20240501 14:05:33.072340 5146 tablet_copy_source_session.cc:205] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:05:33.073626 5276 tablet_copy_service.cc:140] P 1c3fdc5272484ac09ef3e5e36a08f27c: Received BeginTabletCopySession request for tablet 9fd94635797d447cbdc9181ee91f465b from peer 4360b59e3a154688a79ca5f9c0a37bc9 ({username='slave'} at 127.29.124.135:51325)
I20240501 14:05:33.073777 5276 tablet_copy_service.cc:161] P 1c3fdc5272484ac09ef3e5e36a08f27c: Beginning new tablet copy session on tablet 9fd94635797d447cbdc9181ee91f465b from peer 4360b59e3a154688a79ca5f9c0a37bc9 at {username='slave'} at 127.29.124.135:51325: session id = 4360b59e3a154688a79ca5f9c0a37bc9-9fd94635797d447cbdc9181ee91f465b
I20240501 14:05:33.076826 5276 tablet_copy_source_session.cc:205] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:05:33.079392 8241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fd94635797d447cbdc9181ee91f465b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:33.081391 8118 raft_consensus.cc:2932] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:33.082918 8235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9cb784f60774ea7b6e70ccfebf867a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:33.085537 8241 tablet_copy_client.cc:783] T 9fd94635797d447cbdc9181ee91f465b P 4360b59e3a154688a79ca5f9c0a37bc9: tablet copy: Starting download of 0 data blocks...
I20240501 14:05:33.085778 8241 tablet_copy_client.cc:647] T 9fd94635797d447cbdc9181ee91f465b P 4360b59e3a154688a79ca5f9c0a37bc9: tablet copy: Starting download of 1 WAL segments...
I20240501 14:05:33.085901 5255 raft_consensus.cc:2932] T 9b88c43cb8704583a3e65c5c35c205f6 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:33.089323 8241 tablet_copy_client.cc:523] T 9fd94635797d447cbdc9181ee91f465b P 4360b59e3a154688a79ca5f9c0a37bc9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:05:33.091809 4846 catalog_manager.cc:5484] T 9b88c43cb8704583a3e65c5c35c205f6 P 1c3fdc5272484ac09ef3e5e36a08f27c reported cstate change: config changed from index -1 to 4, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:33.092772 8241 tablet_bootstrap.cc:492] T 9fd94635797d447cbdc9181ee91f465b P 4360b59e3a154688a79ca5f9c0a37bc9: Bootstrap starting.
I20240501 14:05:33.097813 8235 tablet_copy_client.cc:783] T f9cb784f60774ea7b6e70ccfebf867a8 P 6dac90f01e744f9c8701a404440a4c77: tablet copy: Starting download of 0 data blocks...
I20240501 14:05:33.098232 8235 tablet_copy_client.cc:647] T f9cb784f60774ea7b6e70ccfebf867a8 P 6dac90f01e744f9c8701a404440a4c77: tablet copy: Starting download of 1 WAL segments...
I20240501 14:05:33.100095 8235 tablet_copy_client.cc:523] T f9cb784f60774ea7b6e70ccfebf867a8 P 6dac90f01e744f9c8701a404440a4c77: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:05:33.105497 8235 tablet_bootstrap.cc:492] T f9cb784f60774ea7b6e70ccfebf867a8 P 6dac90f01e744f9c8701a404440a4c77: Bootstrap starting.
I20240501 14:05:33.114079 4992 raft_consensus.cc:2932] T 9b88c43cb8704583a3e65c5c35c205f6 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:33.120005 5385 consensus_queue.cc:237] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } }
W20240501 14:05:33.127390 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9b88c43cb8704583a3e65c5c35c205f6. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:33.127645 5255 raft_consensus.cc:1261] T 9b88c43cb8704583a3e65c5c35c205f6 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 14:05:33.137981 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9b88c43cb8704583a3e65c5c35c205f6. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:33.139127 8118 consensus_queue.cc:1035] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:33.142033 5255 raft_consensus.cc:2932] T 9b88c43cb8704583a3e65c5c35c205f6 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:33.149317 4846 catalog_manager.cc:5484] T 9b88c43cb8704583a3e65c5c35c205f6 P 1c3fdc5272484ac09ef3e5e36a08f27c reported cstate change: config changed from index 4 to 5, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) added. New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:33.171301 8247 raft_consensus.cc:2932] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:33.176285 4992 raft_consensus.cc:1261] T 9b88c43cb8704583a3e65c5c35c205f6 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:33.177788 8352 consensus_queue.cc:1035] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:33.179864 4992 raft_consensus.cc:2932] T 9b88c43cb8704583a3e65c5c35c205f6 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:33.208644 7287 raft_consensus.cc:1261] T de6c7753c8564107b858876fcfa5924d P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:33.209534 7689 consensus_queue.cc:1035] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:33.214119 8279 raft_consensus.cc:1050] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4: attempting to promote NON_VOTER 6dac90f01e744f9c8701a404440a4c77 to VOTER
I20240501 14:05:33.215075 8279 consensus_queue.cc:237] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } }
I20240501 14:05:33.222687 5384 raft_consensus.cc:1261] T de6c7753c8564107b858876fcfa5924d P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:33.223394 7689 consensus_queue.cc:1035] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:33.230089 8227 tablet_bootstrap.cc:492] T 2c39b53867e047faaf24cc6f87a8100c P 938a24b0e6a040b7966225af26558ff8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:05:33.230754 8227 tablet_bootstrap.cc:492] T 2c39b53867e047faaf24cc6f87a8100c P 938a24b0e6a040b7966225af26558ff8: Bootstrap complete.
I20240501 14:05:33.231298 8227 ts_tablet_manager.cc:1367] T 2c39b53867e047faaf24cc6f87a8100c P 938a24b0e6a040b7966225af26558ff8: Time spent bootstrapping tablet: real 0.429s user 0.058s sys 0.004s
I20240501 14:05:33.239265 4992 raft_consensus.cc:1261] T de6c7753c8564107b858876fcfa5924d P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:33.240295 8285 tablet_bootstrap.cc:492] T 8cc5eec4600640f7a293877c825f8d96 P 19a6d2d8a08142d5bffcde3a37234692: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:05:33.240828 8285 tablet_bootstrap.cc:492] T 8cc5eec4600640f7a293877c825f8d96 P 19a6d2d8a08142d5bffcde3a37234692: Bootstrap complete.
I20240501 14:05:33.241370 8285 ts_tablet_manager.cc:1367] T 8cc5eec4600640f7a293877c825f8d96 P 19a6d2d8a08142d5bffcde3a37234692: Time spent bootstrapping tablet: real 0.460s user 0.076s sys 0.025s
I20240501 14:05:33.241982 8235 tablet_bootstrap.cc:492] T f9cb784f60774ea7b6e70ccfebf867a8 P 6dac90f01e744f9c8701a404440a4c77: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:05:33.242363 8235 tablet_bootstrap.cc:492] T f9cb784f60774ea7b6e70ccfebf867a8 P 6dac90f01e744f9c8701a404440a4c77: Bootstrap complete.
I20240501 14:05:33.242600 8235 ts_tablet_manager.cc:1367] T f9cb784f60774ea7b6e70ccfebf867a8 P 6dac90f01e744f9c8701a404440a4c77: Time spent bootstrapping tablet: real 0.137s user 0.043s sys 0.012s
I20240501 14:05:33.243379 8235 raft_consensus.cc:348] T f9cb784f60774ea7b6e70ccfebf867a8 P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } }
I20240501 14:05:33.243628 8235 raft_consensus.cc:729] T f9cb784f60774ea7b6e70ccfebf867a8 P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6dac90f01e744f9c8701a404440a4c77, State: Initialized, Role: LEARNER
I20240501 14:05:33.243839 8235 consensus_queue.cc:260] T f9cb784f60774ea7b6e70ccfebf867a8 P 6dac90f01e744f9c8701a404440a4c77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } }
I20240501 14:05:33.244215 8281 consensus_queue.cc:1035] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240501 14:05:33.247432 8235 ts_tablet_manager.cc:1398] T f9cb784f60774ea7b6e70ccfebf867a8 P 6dac90f01e744f9c8701a404440a4c77: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:05:33.248265 5146 tablet_copy_service.cc:342] P e9c60449cf3c46f8aca4ab6c535302a4: Request end of tablet copy session 6dac90f01e744f9c8701a404440a4c77-f9cb784f60774ea7b6e70ccfebf867a8 received from {username='slave'} at 127.29.124.136:51401
I20240501 14:05:33.248409 5146 tablet_copy_service.cc:433] P e9c60449cf3c46f8aca4ab6c535302a4: ending tablet copy session 6dac90f01e744f9c8701a404440a4c77-f9cb784f60774ea7b6e70ccfebf867a8 on tablet f9cb784f60774ea7b6e70ccfebf867a8 with peer 6dac90f01e744f9c8701a404440a4c77
I20240501 14:05:33.250291 8241 tablet_bootstrap.cc:492] T 9fd94635797d447cbdc9181ee91f465b P 4360b59e3a154688a79ca5f9c0a37bc9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:05:33.250674 8241 tablet_bootstrap.cc:492] T 9fd94635797d447cbdc9181ee91f465b P 4360b59e3a154688a79ca5f9c0a37bc9: Bootstrap complete.
I20240501 14:05:33.251001 8241 ts_tablet_manager.cc:1367] T 9fd94635797d447cbdc9181ee91f465b P 4360b59e3a154688a79ca5f9c0a37bc9: Time spent bootstrapping tablet: real 0.158s user 0.050s sys 0.001s
I20240501 14:05:33.251775 8241 raft_consensus.cc:348] T 9fd94635797d447cbdc9181ee91f465b P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } }
I20240501 14:05:33.252027 8241 raft_consensus.cc:729] T 9fd94635797d447cbdc9181ee91f465b P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4360b59e3a154688a79ca5f9c0a37bc9, State: Initialized, Role: LEARNER
I20240501 14:05:33.252274 8241 consensus_queue.cc:260] T 9fd94635797d447cbdc9181ee91f465b P 4360b59e3a154688a79ca5f9c0a37bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } }
I20240501 14:05:33.253204 8241 ts_tablet_manager.cc:1398] T 9fd94635797d447cbdc9181ee91f465b P 4360b59e3a154688a79ca5f9c0a37bc9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:05:33.254688 5276 tablet_copy_service.cc:342] P 1c3fdc5272484ac09ef3e5e36a08f27c: Request end of tablet copy session 4360b59e3a154688a79ca5f9c0a37bc9-9fd94635797d447cbdc9181ee91f465b received from {username='slave'} at 127.29.124.135:51325
I20240501 14:05:33.254824 5276 tablet_copy_service.cc:433] P 1c3fdc5272484ac09ef3e5e36a08f27c: ending tablet copy session 4360b59e3a154688a79ca5f9c0a37bc9-9fd94635797d447cbdc9181ee91f465b on tablet 9fd94635797d447cbdc9181ee91f465b with peer 4360b59e3a154688a79ca5f9c0a37bc9
I20240501 14:05:33.255128 8205 tablet_bootstrap.cc:492] T 1a7bbfc15a76450bbc9197037e856e08 P 19a6d2d8a08142d5bffcde3a37234692: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:05:33.256703 8205 tablet_bootstrap.cc:492] T 1a7bbfc15a76450bbc9197037e856e08 P 19a6d2d8a08142d5bffcde3a37234692: Bootstrap complete.
I20240501 14:05:33.257493 8227 raft_consensus.cc:348] T 2c39b53867e047faaf24cc6f87a8100c P 938a24b0e6a040b7966225af26558ff8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }
I20240501 14:05:33.258432 8227 raft_consensus.cc:729] T 2c39b53867e047faaf24cc6f87a8100c P 938a24b0e6a040b7966225af26558ff8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 938a24b0e6a040b7966225af26558ff8, State: Initialized, Role: LEARNER
I20240501 14:05:33.259081 8227 consensus_queue.cc:260] T 2c39b53867e047faaf24cc6f87a8100c P 938a24b0e6a040b7966225af26558ff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }
I20240501 14:05:33.259344 8285 raft_consensus.cc:348] T 8cc5eec4600640f7a293877c825f8d96 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } }
I20240501 14:05:33.260069 8285 raft_consensus.cc:729] T 8cc5eec4600640f7a293877c825f8d96 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 19a6d2d8a08142d5bffcde3a37234692, State: Initialized, Role: LEARNER
I20240501 14:05:33.260639 8285 consensus_queue.cc:260] T 8cc5eec4600640f7a293877c825f8d96 P 19a6d2d8a08142d5bffcde3a37234692 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } }
I20240501 14:05:33.262277 8205 ts_tablet_manager.cc:1367] T 1a7bbfc15a76450bbc9197037e856e08 P 19a6d2d8a08142d5bffcde3a37234692: Time spent bootstrapping tablet: real 0.472s user 0.056s sys 0.009s
I20240501 14:05:33.263309 8205 raft_consensus.cc:348] T 1a7bbfc15a76450bbc9197037e856e08 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } }
I20240501 14:05:33.263535 8205 raft_consensus.cc:729] T 1a7bbfc15a76450bbc9197037e856e08 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 19a6d2d8a08142d5bffcde3a37234692, State: Initialized, Role: LEARNER
I20240501 14:05:33.263758 8205 consensus_queue.cc:260] T 1a7bbfc15a76450bbc9197037e856e08 P 19a6d2d8a08142d5bffcde3a37234692 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } }
I20240501 14:05:33.265375 5126 consensus_queue.cc:237] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }
I20240501 14:05:33.303893 7286 raft_consensus.cc:1261] T 9d32e0707d244f2c9bbcb57f62b09f31 P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:33.304626 8149 consensus_queue.cc:1035] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:33.310726 8281 raft_consensus.cc:2932] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:33.314078 4995 raft_consensus.cc:2932] T de6c7753c8564107b858876fcfa5924d P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:33.322540 7287 raft_consensus.cc:2932] T de6c7753c8564107b858876fcfa5924d P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:33.325418 5255 raft_consensus.cc:1261] T 3d6859d53fd747a5a5e94722626fa522 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:05:33.326086 8149 consensus_queue.cc:1035] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:05:33.328101 4853 catalog_manager.cc:5484] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c reported cstate change: config changed from index 4 to 5, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:05:33.335819 5385 raft_consensus.cc:1261] T 9d32e0707d244f2c9bbcb57f62b09f31 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:33.336668 8149 consensus_queue.cc:1035] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:33.347555 5254 consensus_queue.cc:237] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } }
W20240501 14:05:33.347803 4929 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.124.130:34377, user_credentials={real_user=slave}} blocked reactor thread for 145806us
I20240501 14:05:33.349846 7287 raft_consensus.cc:1261] T de6c7753c8564107b858876fcfa5924d P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:33.350545 8281 consensus_queue.cc:1035] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:33.351279 5254 consensus_queue.cc:237] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } }
I20240501 14:05:33.353291 8277 raft_consensus.cc:2932] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:33.357164 7285 raft_consensus.cc:2932] T de6c7753c8564107b858876fcfa5924d P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:33.360633 4838 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet de6c7753c8564107b858876fcfa5924d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:05:33.362875 4991 raft_consensus.cc:1261] T 3d6859d53fd747a5a5e94722626fa522 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:05:33.362738 8175 raft_consensus.cc:2932] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:33.363448 4991 raft_consensus.cc:1261] T 9d32e0707d244f2c9bbcb57f62b09f31 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:33.363566 4853 catalog_manager.cc:5484] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c reported cstate change: config changed from index 5 to 6, VOTER 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129) evicted. New cstate: current_term: 1 leader_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:05:33.367288 8149 consensus_queue.cc:1035] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:05:33.368135 4973 tablet_service.cc:1514] Processing DeleteTablet for tablet de6c7753c8564107b858876fcfa5924d with delete_type TABLET_DATA_TOMBSTONED (TS 04088ee3bea440a7bd5ff772152b9410 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41194
I20240501 14:05:33.347057 5384 raft_consensus.cc:2932] T de6c7753c8564107b858876fcfa5924d P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:33.376335 8285 ts_tablet_manager.cc:1398] T 8cc5eec4600640f7a293877c825f8d96 P 19a6d2d8a08142d5bffcde3a37234692: Time spent starting tablet: real 0.135s user 0.010s sys 0.024s
I20240501 14:05:33.377125 5015 tablet_copy_service.cc:342] P 04088ee3bea440a7bd5ff772152b9410: Request end of tablet copy session 19a6d2d8a08142d5bffcde3a37234692-8cc5eec4600640f7a293877c825f8d96 received from {username='slave'} at 127.29.124.134:36629
I20240501 14:05:33.377234 5015 tablet_copy_service.cc:433] P 04088ee3bea440a7bd5ff772152b9410: ending tablet copy session 19a6d2d8a08142d5bffcde3a37234692-8cc5eec4600640f7a293877c825f8d96 on tablet 8cc5eec4600640f7a293877c825f8d96 with peer 19a6d2d8a08142d5bffcde3a37234692
W20240501 14:05:33.383601 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 23ca2bf1e59f44fcac8a48aff8d056b1. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:33.372655 5255 raft_consensus.cc:2932] T 3d6859d53fd747a5a5e94722626fa522 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:33.393368 7156 raft_consensus.cc:1203] T 9fd94635797d447cbdc9181ee91f465b P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:05:33.402474 5383 raft_consensus.cc:1261] T 23ca2bf1e59f44fcac8a48aff8d056b1 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:05:33.403198 8277 consensus_queue.cc:1035] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:05:33.404512 8205 ts_tablet_manager.cc:1398] T 1a7bbfc15a76450bbc9197037e856e08 P 19a6d2d8a08142d5bffcde3a37234692: Time spent starting tablet: real 0.142s user 0.014s sys 0.021s
I20240501 14:05:33.405615 5276 tablet_copy_service.cc:342] P 1c3fdc5272484ac09ef3e5e36a08f27c: Request end of tablet copy session 19a6d2d8a08142d5bffcde3a37234692-1a7bbfc15a76450bbc9197037e856e08 received from {username='slave'} at 127.29.124.134:43435
I20240501 14:05:33.405766 5276 tablet_copy_service.cc:433] P 1c3fdc5272484ac09ef3e5e36a08f27c: ending tablet copy session 19a6d2d8a08142d5bffcde3a37234692-1a7bbfc15a76450bbc9197037e856e08 on tablet 1a7bbfc15a76450bbc9197037e856e08 with peer 19a6d2d8a08142d5bffcde3a37234692
I20240501 14:05:33.408008 8386 tablet_replica.cc:330] T de6c7753c8564107b858876fcfa5924d P 04088ee3bea440a7bd5ff772152b9410: stopping tablet replica
I20240501 14:05:33.408645 8386 raft_consensus.cc:2229] T de6c7753c8564107b858876fcfa5924d P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:05:33.409037 8386 raft_consensus.cc:2258] T de6c7753c8564107b858876fcfa5924d P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:05:33.412155 8386 ts_tablet_manager.cc:1875] T de6c7753c8564107b858876fcfa5924d P 04088ee3bea440a7bd5ff772152b9410: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240501 14:05:33.412355 5060 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.124.129:44033, user_credentials={real_user=slave}} blocked reactor thread for 47697.3us
I20240501 14:05:33.415553 7156 raft_consensus.cc:1203] T ac4b234f794149e4b6def7a3b358942d P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:05:33.418166 8277 raft_consensus.cc:2932] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } }
I20240501 14:05:33.421991 5382 raft_consensus.cc:2932] T 23ca2bf1e59f44fcac8a48aff8d056b1 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } }
I20240501 14:05:33.428639 4846 catalog_manager.cc:5484] T 23ca2bf1e59f44fcac8a48aff8d056b1 P be56773105e744919601127a94e98adf reported cstate change: config changed from index -1 to 4, NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) added. New cstate: current_term: 1 leader_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } }
I20240501 14:05:33.442610 5125 raft_consensus.cc:1261] T 23ca2bf1e59f44fcac8a48aff8d056b1 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:05:33.447940 4846 catalog_manager.cc:5484] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 reported cstate change: config changed from index -1 to 4, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New cstate: current_term: 1 leader_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:05:33.451191 8149 consensus_queue.cc:1035] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:33.453634 8277 consensus_queue.cc:1035] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:05:33.456799 5386 raft_consensus.cc:1261] T de6c7753c8564107b858876fcfa5924d P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:33.457732 8277 consensus_queue.cc:1035] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:33.459127 5386 raft_consensus.cc:2932] T de6c7753c8564107b858876fcfa5924d P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:33.474898 8279 raft_consensus.cc:2932] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:33.479000 8235 ts_tablet_manager.cc:902] T 9afe3b20a43d4b5bb800af39f945948d P 6dac90f01e744f9c8701a404440a4c77: Initiating tablet copy from peer e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377)
I20240501 14:05:33.481308 7286 raft_consensus.cc:2932] T 9d32e0707d244f2c9bbcb57f62b09f31 P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:33.485864 8235 tablet_copy_client.cc:308] T 9afe3b20a43d4b5bb800af39f945948d P 6dac90f01e744f9c8701a404440a4c77: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.130:34377
I20240501 14:05:33.485919 4992 raft_consensus.cc:2932] T 3d6859d53fd747a5a5e94722626fa522 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:33.489609 5384 raft_consensus.cc:2932] T 9d32e0707d244f2c9bbcb57f62b09f31 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:33.490583 5126 raft_consensus.cc:2932] T 23ca2bf1e59f44fcac8a48aff8d056b1 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } }
I20240501 14:05:33.503608 4853 catalog_manager.cc:5484] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 reported cstate change: config changed from index 4 to 5, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:05:33.514494 8205 ts_tablet_manager.cc:902] T f4d2b8c005b64bbb8e6621da068d15e5 P 19a6d2d8a08142d5bffcde3a37234692: Initiating tablet copy from peer be56773105e744919601127a94e98adf (127.29.124.132:35459)
I20240501 14:05:33.516870 5125 consensus_queue.cc:237] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } }
I20240501 14:05:33.526214 8227 ts_tablet_manager.cc:1398] T 2c39b53867e047faaf24cc6f87a8100c P 938a24b0e6a040b7966225af26558ff8: Time spent starting tablet: real 0.295s user 0.100s sys 0.116s
I20240501 14:05:33.526312 5146 tablet_copy_service.cc:140] P e9c60449cf3c46f8aca4ab6c535302a4: Received BeginTabletCopySession request for tablet 9afe3b20a43d4b5bb800af39f945948d from peer 6dac90f01e744f9c8701a404440a4c77 ({username='slave'} at 127.29.124.136:51401)
I20240501 14:05:33.526471 5146 tablet_copy_service.cc:161] P e9c60449cf3c46f8aca4ab6c535302a4: Beginning new tablet copy session on tablet 9afe3b20a43d4b5bb800af39f945948d from peer 6dac90f01e744f9c8701a404440a4c77 at {username='slave'} at 127.29.124.136:51401: session id = 6dac90f01e744f9c8701a404440a4c77-9afe3b20a43d4b5bb800af39f945948d
I20240501 14:05:33.529212 5146 tablet_copy_source_session.cc:205] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4: Tablet Copy: opened 0 blocks and 1 log segments
W20240501 14:05:33.531922 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3d6859d53fd747a5a5e94722626fa522. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:33.534180 5406 tablet_copy_service.cc:342] P be56773105e744919601127a94e98adf: Request end of tablet copy session 938a24b0e6a040b7966225af26558ff8-2c39b53867e047faaf24cc6f87a8100c received from {username='slave'} at 127.29.124.133:50583
I20240501 14:05:33.534353 5406 tablet_copy_service.cc:433] P be56773105e744919601127a94e98adf: ending tablet copy session 938a24b0e6a040b7966225af26558ff8-2c39b53867e047faaf24cc6f87a8100c on tablet 2c39b53867e047faaf24cc6f87a8100c with peer 938a24b0e6a040b7966225af26558ff8
I20240501 14:05:33.535051 8205 tablet_copy_client.cc:308] T f4d2b8c005b64bbb8e6621da068d15e5 P 19a6d2d8a08142d5bffcde3a37234692: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.132:35459
I20240501 14:05:33.536262 8227 ts_tablet_manager.cc:902] T 9b88c43cb8704583a3e65c5c35c205f6 P 938a24b0e6a040b7966225af26558ff8: Initiating tablet copy from peer be56773105e744919601127a94e98adf (127.29.124.132:35459)
I20240501 14:05:33.541241 8235 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9afe3b20a43d4b5bb800af39f945948d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:33.543304 7285 raft_consensus.cc:1261] T 9d32e0707d244f2c9bbcb57f62b09f31 P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:33.544662 8149 consensus_queue.cc:1035] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:33.547986 8149 raft_consensus.cc:2932] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:33.550715 7285 raft_consensus.cc:2932] T 9d32e0707d244f2c9bbcb57f62b09f31 P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:33.552559 4991 consensus_queue.cc:237] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } }
I20240501 14:05:33.553529 5385 raft_consensus.cc:1261] T 9d32e0707d244f2c9bbcb57f62b09f31 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:33.554868 8279 consensus_queue.cc:1035] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:33.555976 8235 tablet_copy_client.cc:783] T 9afe3b20a43d4b5bb800af39f945948d P 6dac90f01e744f9c8701a404440a4c77: tablet copy: Starting download of 0 data blocks...
I20240501 14:05:33.556814 8241 ts_tablet_manager.cc:902] T 9b88c43cb8704583a3e65c5c35c205f6 P 4360b59e3a154688a79ca5f9c0a37bc9: Initiating tablet copy from peer be56773105e744919601127a94e98adf (127.29.124.132:35459)
I20240501 14:05:33.557065 8235 tablet_copy_client.cc:647] T 9afe3b20a43d4b5bb800af39f945948d P 6dac90f01e744f9c8701a404440a4c77: tablet copy: Starting download of 1 WAL segments...
I20240501 14:05:33.560384 8386 ts_tablet_manager.cc:1888] T de6c7753c8564107b858876fcfa5924d P 04088ee3bea440a7bd5ff772152b9410: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:05:33.559991 4848 catalog_manager.cc:5484] T 9d32e0707d244f2c9bbcb57f62b09f31 P 6dac90f01e744f9c8701a404440a4c77 reported cstate change: config changed from index 5 to 6, VOTER 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129) evicted. New cstate: current_term: 1 leader_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:33.562175 8386 log.cc:1196] T de6c7753c8564107b858876fcfa5924d P 04088ee3bea440a7bd5ff772152b9410: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/wal/wals/de6c7753c8564107b858876fcfa5924d
I20240501 14:05:33.563151 4839 catalog_manager.cc:4830] TS 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129:44033): tablet de6c7753c8564107b858876fcfa5924d (table rebalance_test_table_6 [id=3c1513e944ee4b518373c4efa5e200cc]) successfully deleted
I20240501 14:05:33.563686 5385 raft_consensus.cc:2932] T 9d32e0707d244f2c9bbcb57f62b09f31 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:33.566162 4973 tablet_service.cc:1514] Processing DeleteTablet for tablet 9d32e0707d244f2c9bbcb57f62b09f31 with delete_type TABLET_DATA_TOMBSTONED (TS 04088ee3bea440a7bd5ff772152b9410 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41194
I20240501 14:05:33.566644 8235 tablet_copy_client.cc:523] T 9afe3b20a43d4b5bb800af39f945948d P 6dac90f01e744f9c8701a404440a4c77: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:05:33.566709 8386 tablet_replica.cc:330] T 9d32e0707d244f2c9bbcb57f62b09f31 P 04088ee3bea440a7bd5ff772152b9410: stopping tablet replica
I20240501 14:05:33.569614 4839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9d32e0707d244f2c9bbcb57f62b09f31 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:05:33.571334 4992 consensus_queue.cc:237] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }
I20240501 14:05:33.571789 8386 raft_consensus.cc:2229] T 9d32e0707d244f2c9bbcb57f62b09f31 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:05:33.572343 8386 pending_rounds.cc:62] T 9d32e0707d244f2c9bbcb57f62b09f31 P 04088ee3bea440a7bd5ff772152b9410: Trying to abort 1 pending ops.
I20240501 14:05:33.572503 8386 pending_rounds.cc:68] T 9d32e0707d244f2c9bbcb57f62b09f31 P 04088ee3bea440a7bd5ff772152b9410: Aborting op as it isn't in flight: id { term: 1 index: 5 } timestamp: 7022888276846551040 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "9d32e0707d244f2c9bbcb57f62b09f31" old_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } } new_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } } }
I20240501 14:05:33.574685 8386 raft_consensus.cc:2894] T 9d32e0707d244f2c9bbcb57f62b09f31 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Aborting config change with OpId 1.5: Aborted: Op aborted
I20240501 14:05:33.575034 8386 raft_consensus.cc:2866] T 9d32e0707d244f2c9bbcb57f62b09f31 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 14:05:33.575040 5255 consensus_queue.cc:237] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }
I20240501 14:05:33.575416 8386 raft_consensus.cc:2258] T 9d32e0707d244f2c9bbcb57f62b09f31 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:05:33.578651 8235 tablet_bootstrap.cc:492] T 9afe3b20a43d4b5bb800af39f945948d P 6dac90f01e744f9c8701a404440a4c77: Bootstrap starting.
I20240501 14:05:33.579842 8386 ts_tablet_manager.cc:1875] T 9d32e0707d244f2c9bbcb57f62b09f31 P 04088ee3bea440a7bd5ff772152b9410: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:05:33.581292 7156 raft_consensus.cc:1261] T fee16905387d428aa02cd2070c86c790 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 LEARNER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:33.588645 5406 tablet_copy_service.cc:140] P be56773105e744919601127a94e98adf: Received BeginTabletCopySession request for tablet f4d2b8c005b64bbb8e6621da068d15e5 from peer 19a6d2d8a08142d5bffcde3a37234692 ({username='slave'} at 127.29.124.134:56125)
I20240501 14:05:33.588819 5406 tablet_copy_service.cc:161] P be56773105e744919601127a94e98adf: Beginning new tablet copy session on tablet f4d2b8c005b64bbb8e6621da068d15e5 from peer 19a6d2d8a08142d5bffcde3a37234692 at {username='slave'} at 127.29.124.134:56125: session id = 19a6d2d8a08142d5bffcde3a37234692-f4d2b8c005b64bbb8e6621da068d15e5
I20240501 14:05:33.581974 8210 consensus_queue.cc:1035] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:33.591666 5406 tablet_copy_source_session.cc:205] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:05:33.593130 8205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4d2b8c005b64bbb8e6621da068d15e5. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 14:05:33.594166 4930 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:33.596468 5384 raft_consensus.cc:1261] T fee16905387d428aa02cd2070c86c790 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:33.597349 8241 tablet_copy_client.cc:308] T 9b88c43cb8704583a3e65c5c35c205f6 P 4360b59e3a154688a79ca5f9c0a37bc9: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.132:35459
I20240501 14:05:33.599458 8205 tablet_copy_client.cc:783] T f4d2b8c005b64bbb8e6621da068d15e5 P 19a6d2d8a08142d5bffcde3a37234692: tablet copy: Starting download of 0 data blocks...
I20240501 14:05:33.599673 8205 tablet_copy_client.cc:647] T f4d2b8c005b64bbb8e6621da068d15e5 P 19a6d2d8a08142d5bffcde3a37234692: tablet copy: Starting download of 1 WAL segments...
I20240501 14:05:33.601122 8205 tablet_copy_client.cc:523] T f4d2b8c005b64bbb8e6621da068d15e5 P 19a6d2d8a08142d5bffcde3a37234692: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:05:33.602277 5126 raft_consensus.cc:1261] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:33.604388 8205 tablet_bootstrap.cc:492] T f4d2b8c005b64bbb8e6621da068d15e5 P 19a6d2d8a08142d5bffcde3a37234692: Bootstrap starting.
I20240501 14:05:33.605715 7025 raft_consensus.cc:1261] T 8cc5eec4600640f7a293877c825f8d96 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 LEARNER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:33.606701 8210 consensus_queue.cc:1035] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:33.611507 8227 tablet_copy_client.cc:308] T 9b88c43cb8704583a3e65c5c35c205f6 P 938a24b0e6a040b7966225af26558ff8: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.132:35459
I20240501 14:05:33.612234 5406 tablet_copy_service.cc:140] P be56773105e744919601127a94e98adf: Received BeginTabletCopySession request for tablet 9b88c43cb8704583a3e65c5c35c205f6 from peer 938a24b0e6a040b7966225af26558ff8 ({username='slave'} at 127.29.124.133:50583)
I20240501 14:05:33.612376 5406 tablet_copy_service.cc:161] P be56773105e744919601127a94e98adf: Beginning new tablet copy session on tablet 9b88c43cb8704583a3e65c5c35c205f6 from peer 938a24b0e6a040b7966225af26558ff8 at {username='slave'} at 127.29.124.133:50583: session id = 938a24b0e6a040b7966225af26558ff8-9b88c43cb8704583a3e65c5c35c205f6
I20240501 14:05:33.612390 8222 consensus_queue.cc:1035] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:33.612959 8222 consensus_queue.cc:1035] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:33.615278 5406 tablet_copy_source_session.cc:205] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:05:33.620079 7286 raft_consensus.cc:1203] T f9cb784f60774ea7b6e70ccfebf867a8 P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:05:33.622081 8227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b88c43cb8704583a3e65c5c35c205f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:33.631793 5385 raft_consensus.cc:1261] T 71660575d94748d8a9329f909a42a0cd P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:05:33.632231 8227 tablet_copy_client.cc:783] T 9b88c43cb8704583a3e65c5c35c205f6 P 938a24b0e6a040b7966225af26558ff8: tablet copy: Starting download of 0 data blocks...
I20240501 14:05:33.632484 8227 tablet_copy_client.cc:647] T 9b88c43cb8704583a3e65c5c35c205f6 P 938a24b0e6a040b7966225af26558ff8: tablet copy: Starting download of 1 WAL segments...
I20240501 14:05:33.632462 8281 consensus_queue.cc:1035] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:05:33.634153 8222 raft_consensus.cc:1013] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEADER]: attempt to promote peer 19a6d2d8a08142d5bffcde3a37234692: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 14:05:33.635535 5125 raft_consensus.cc:1261] T 8cc5eec4600640f7a293877c825f8d96 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:33.636412 8363 consensus_queue.cc:1035] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:33.636523 5386 raft_consensus.cc:1261] T 8cc5eec4600640f7a293877c825f8d96 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 14:05:33.637224 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8cc5eec4600640f7a293877c825f8d96. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:33.643399 4995 raft_consensus.cc:1261] T 71660575d94748d8a9329f909a42a0cd P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:05:33.644191 8227 tablet_copy_client.cc:523] T 9b88c43cb8704583a3e65c5c35c205f6 P 938a24b0e6a040b7966225af26558ff8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:05:33.644070 8363 consensus_queue.cc:1035] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:33.644618 8281 consensus_queue.cc:1035] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:05:33.644933 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 71660575d94748d8a9329f909a42a0cd. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:33.644932 8362 raft_consensus.cc:1013] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEADER]: attempt to promote peer 4360b59e3a154688a79ca5f9c0a37bc9: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 14:05:33.656508 8222 raft_consensus.cc:2932] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:33.661816 7156 raft_consensus.cc:2932] T fee16905387d428aa02cd2070c86c790 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:33.663583 4992 consensus_queue.cc:237] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } }
I20240501 14:05:33.664111 4853 catalog_manager.cc:5484] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 reported cstate change: config changed from index 5 to 6, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New cstate: current_term: 1 leader_uuid: "04088ee3bea440a7bd5ff772152b9410" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:05:33.667657 5126 raft_consensus.cc:2932] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:33.675029 8227 tablet_bootstrap.cc:492] T 9b88c43cb8704583a3e65c5c35c205f6 P 938a24b0e6a040b7966225af26558ff8: Bootstrap starting.
I20240501 14:05:33.676271 8210 raft_consensus.cc:2932] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:33.682359 4853 catalog_manager.cc:5484] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 reported cstate change: config changed from index 4 to 5, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New cstate: current_term: 1 leader_uuid: "04088ee3bea440a7bd5ff772152b9410" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:05:33.683977 6892 raft_consensus.cc:1203] T 2c39b53867e047faaf24cc6f87a8100c P 938a24b0e6a040b7966225af26558ff8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:05:33.684373 5386 raft_consensus.cc:2932] T 8cc5eec4600640f7a293877c825f8d96 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:33.684012 7025 raft_consensus.cc:2932] T 8cc5eec4600640f7a293877c825f8d96 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
W20240501 14:05:33.689652 8290 log.cc:924] Time spent T fee16905387d428aa02cd2070c86c790 P 938a24b0e6a040b7966225af26558ff8: Append to log took a long time: real 0.088s user 0.003s sys 0.000s
I20240501 14:05:33.690812 8290 tablet_bootstrap.cc:492] T fee16905387d428aa02cd2070c86c790 P 938a24b0e6a040b7966225af26558ff8: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:05:33.691256 8290 tablet_bootstrap.cc:492] T fee16905387d428aa02cd2070c86c790 P 938a24b0e6a040b7966225af26558ff8: Bootstrap complete.
I20240501 14:05:33.691488 8290 ts_tablet_manager.cc:1367] T fee16905387d428aa02cd2070c86c790 P 938a24b0e6a040b7966225af26558ff8: Time spent bootstrapping tablet: real 0.716s user 0.061s sys 0.005s
I20240501 14:05:33.692373 8290 raft_consensus.cc:348] T fee16905387d428aa02cd2070c86c790 P 938a24b0e6a040b7966225af26558ff8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }
I20240501 14:05:33.692625 8290 raft_consensus.cc:729] T fee16905387d428aa02cd2070c86c790 P 938a24b0e6a040b7966225af26558ff8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 938a24b0e6a040b7966225af26558ff8, State: Initialized, Role: LEARNER
I20240501 14:05:33.692828 8290 consensus_queue.cc:260] T fee16905387d428aa02cd2070c86c790 P 938a24b0e6a040b7966225af26558ff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }
I20240501 14:05:33.694193 8281 raft_consensus.cc:2932] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:33.697067 8222 raft_consensus.cc:1050] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410: attempting to promote NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 to VOTER
I20240501 14:05:33.698185 5382 raft_consensus.cc:2932] T 71660575d94748d8a9329f909a42a0cd P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:33.701368 5124 raft_consensus.cc:2932] T 8cc5eec4600640f7a293877c825f8d96 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:33.703452 4995 raft_consensus.cc:2932] T 71660575d94748d8a9329f909a42a0cd P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:33.710459 8222 consensus_queue.cc:237] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }
I20240501 14:05:33.712244 8290 ts_tablet_manager.cc:1398] T fee16905387d428aa02cd2070c86c790 P 938a24b0e6a040b7966225af26558ff8: Time spent starting tablet: real 0.021s user 0.003s sys 0.000s
I20240501 14:05:33.712226 4848 catalog_manager.cc:5484] T 71660575d94748d8a9329f909a42a0cd P be56773105e744919601127a94e98adf reported cstate change: config changed from index -1 to 4, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New cstate: current_term: 1 leader_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:33.721839 8409 ts_tablet_manager.cc:902] T 3d6859d53fd747a5a5e94722626fa522 P 938a24b0e6a040b7966225af26558ff8: Initiating tablet copy from peer e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377)
I20240501 14:05:33.723413 5015 tablet_copy_service.cc:342] P 04088ee3bea440a7bd5ff772152b9410: Request end of tablet copy session 938a24b0e6a040b7966225af26558ff8-fee16905387d428aa02cd2070c86c790 received from {username='slave'} at 127.29.124.133:49901
I20240501 14:05:33.723532 5015 tablet_copy_service.cc:433] P 04088ee3bea440a7bd5ff772152b9410: ending tablet copy session 938a24b0e6a040b7966225af26558ff8-fee16905387d428aa02cd2070c86c790 on tablet fee16905387d428aa02cd2070c86c790 with peer 938a24b0e6a040b7966225af26558ff8
W20240501 14:05:33.726364 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8cc5eec4600640f7a293877c825f8d96. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:33.726486 5126 raft_consensus.cc:1261] T 8cc5eec4600640f7a293877c825f8d96 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:33.726989 8222 consensus_queue.cc:1035] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:33.727295 7025 raft_consensus.cc:1261] T 8cc5eec4600640f7a293877c825f8d96 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 LEARNER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:33.729244 8375 consensus_queue.cc:1035] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:33.735597 8222 raft_consensus.cc:2932] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:33.739377 5384 raft_consensus.cc:2932] T fee16905387d428aa02cd2070c86c790 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:33.744148 5382 raft_consensus.cc:1261] T 8cc5eec4600640f7a293877c825f8d96 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:33.748555 7025 raft_consensus.cc:2932] T 8cc5eec4600640f7a293877c825f8d96 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:33.753903 8363 consensus_queue.cc:1035] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:33.753882 5126 raft_consensus.cc:2932] T 8cc5eec4600640f7a293877c825f8d96 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:33.755559 5382 raft_consensus.cc:2932] T 8cc5eec4600640f7a293877c825f8d96 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:33.759460 4848 catalog_manager.cc:5484] T 8cc5eec4600640f7a293877c825f8d96 P e9c60449cf3c46f8aca4ab6c535302a4 reported cstate change: config changed from index 5 to 6, 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "04088ee3bea440a7bd5ff772152b9410" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:33.771930 4995 consensus_queue.cc:237] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } }
I20240501 14:05:33.775566 7025 raft_consensus.cc:1261] T 8cc5eec4600640f7a293877c825f8d96 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:05:33.775640 5126 raft_consensus.cc:1261] T 8cc5eec4600640f7a293877c825f8d96 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:05:33.776216 8375 consensus_queue.cc:1035] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:05:33.776686 8375 consensus_queue.cc:1035] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:05:33.782850 8363 raft_consensus.cc:1050] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410: attempting to promote NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 to VOTER
I20240501 14:05:33.783692 8363 consensus_queue.cc:237] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } }
I20240501 14:05:33.807336 5254 raft_consensus.cc:1261] T ba6cba753e934e719c730593c0706346 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:05:33.807642 5126 raft_consensus.cc:1261] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:05:33.808187 8375 consensus_queue.cc:1035] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:05:33.808369 8210 consensus_queue.cc:1035] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:05:33.810487 4931 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fee16905387d428aa02cd2070c86c790. This is attempt 1: this message will repeat every 5th retry.
W20240501 14:05:33.813035 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ba6cba753e934e719c730593c0706346. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:33.813360 7155 raft_consensus.cc:1261] T fee16905387d428aa02cd2070c86c790 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 LEARNER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:05:33.813791 6892 raft_consensus.cc:1261] T fee16905387d428aa02cd2070c86c790 P 938a24b0e6a040b7966225af26558ff8 [term 1 LEARNER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:05:33.814536 8362 consensus_queue.cc:1035] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:33.815024 8362 consensus_queue.cc:1035] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:05:33.815156 8281 raft_consensus.cc:1050] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c: attempting to promote NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 to VOTER
I20240501 14:05:33.815909 8281 consensus_queue.cc:237] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } }
I20240501 14:05:33.817425 6892 raft_consensus.cc:858] T fee16905387d428aa02cd2070c86c790 P 938a24b0e6a040b7966225af26558ff8 [term 1 LEARNER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }.
Pending config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } }:
New config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } }
W20240501 14:05:33.817552 6892 raft_consensus.cc:1565] T fee16905387d428aa02cd2070c86c790 P 938a24b0e6a040b7966225af26558ff8 [term 1 LEARNER]: Could not prepare op 'term: 1 index: 7' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }.
Pending config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } }:
New config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } }
I20240501 14:05:33.819201 8210 raft_consensus.cc:2932] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:33.821976 7156 raft_consensus.cc:2932] T fee16905387d428aa02cd2070c86c790 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:33.831102 8383 raft_consensus.cc:1050] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4: attempting to promote NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 to VOTER
I20240501 14:05:33.831956 8383 consensus_queue.cc:237] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } }
I20240501 14:05:33.836558 4846 catalog_manager.cc:5484] T fee16905387d428aa02cd2070c86c790 P 4360b59e3a154688a79ca5f9c0a37bc9 reported cstate change: config changed from index 6 to 7, 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "04088ee3bea440a7bd5ff772152b9410" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:33.841706 4995 raft_consensus.cc:1261] T ac4b234f794149e4b6def7a3b358942d P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:33.842329 8383 consensus_queue.cc:1035] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:33.845829 7156 raft_consensus.cc:1261] T ac4b234f794149e4b6def7a3b358942d P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 LEARNER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:33.846307 8383 consensus_queue.cc:1035] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:33.848060 5382 raft_consensus.cc:1261] T fee16905387d428aa02cd2070c86c790 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:05:33.849205 8362 consensus_queue.cc:1035] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:05:33.850028 5386 raft_consensus.cc:1261] T 8cc5eec4600640f7a293877c825f8d96 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:05:33.851704 5126 raft_consensus.cc:1261] T ba6cba753e934e719c730593c0706346 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:05:33.852312 8362 consensus_queue.cc:1035] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:05:33.853629 8222 raft_consensus.cc:2932] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:33.858242 8362 consensus_queue.cc:1035] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:05:33.859647 8285 ts_tablet_manager.cc:902] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 19a6d2d8a08142d5bffcde3a37234692: Initiating tablet copy from peer 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209)
I20240501 14:05:33.859973 5386 consensus_queue.cc:237] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } }
I20240501 14:05:33.862111 5384 raft_consensus.cc:1261] T ac4b234f794149e4b6def7a3b358942d P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:33.862458 5382 raft_consensus.cc:2932] T fee16905387d428aa02cd2070c86c790 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:33.862836 8383 consensus_queue.cc:1035] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:33.868636 5382 raft_consensus.cc:2932] T 8cc5eec4600640f7a293877c825f8d96 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:33.868794 8375 raft_consensus.cc:2932] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:33.874817 5255 raft_consensus.cc:2932] T ba6cba753e934e719c730593c0706346 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:33.880738 5125 raft_consensus.cc:2932] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:33.874029 6892 raft_consensus.cc:2932] T fee16905387d428aa02cd2070c86c790 P 938a24b0e6a040b7966225af26558ff8 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:33.884675 5124 raft_consensus.cc:1261] T 9fd94635797d447cbdc9181ee91f465b P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:33.885427 7155 raft_consensus.cc:1261] T 9fd94635797d447cbdc9181ee91f465b P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 LEARNER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:33.885700 8281 consensus_queue.cc:1035] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:33.886245 7689 consensus_queue.cc:1035] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:33.888948 4995 raft_consensus.cc:1261] T 9fd94635797d447cbdc9181ee91f465b P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:33.889592 7689 consensus_queue.cc:1035] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:33.891626 4846 catalog_manager.cc:5484] T ba6cba753e934e719c730593c0706346 P 1c3fdc5272484ac09ef3e5e36a08f27c reported cstate change: config changed from index -1 to 4, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) added. New cstate: current_term: 1 leader_uuid: "04088ee3bea440a7bd5ff772152b9410" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:33.900342 8386 ts_tablet_manager.cc:1888] T 9d32e0707d244f2c9bbcb57f62b09f31 P 04088ee3bea440a7bd5ff772152b9410: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:05:33.900496 8386 log.cc:1196] T 9d32e0707d244f2c9bbcb57f62b09f31 P 04088ee3bea440a7bd5ff772152b9410: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/wal/wals/9d32e0707d244f2c9bbcb57f62b09f31
I20240501 14:05:33.902122 5126 raft_consensus.cc:2932] T ba6cba753e934e719c730593c0706346 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:33.907078 8279 raft_consensus.cc:2932] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:33.912314 8277 raft_consensus.cc:2932] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:33.914335 8227 tablet_bootstrap.cc:492] T 9b88c43cb8704583a3e65c5c35c205f6 P 938a24b0e6a040b7966225af26558ff8: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:05:33.914750 8227 tablet_bootstrap.cc:492] T 9b88c43cb8704583a3e65c5c35c205f6 P 938a24b0e6a040b7966225af26558ff8: Bootstrap complete.
I20240501 14:05:33.914491 4848 catalog_manager.cc:5484] T 8cc5eec4600640f7a293877c825f8d96 P be56773105e744919601127a94e98adf reported cstate change: config changed from index 6 to 7, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) evicted. New cstate: current_term: 1 leader_uuid: "04088ee3bea440a7bd5ff772152b9410" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:33.915006 8227 ts_tablet_manager.cc:1367] T 9b88c43cb8704583a3e65c5c35c205f6 P 938a24b0e6a040b7966225af26558ff8: Time spent bootstrapping tablet: real 0.240s user 0.046s sys 0.002s
I20240501 14:05:33.915757 8227 raft_consensus.cc:348] T 9b88c43cb8704583a3e65c5c35c205f6 P 938a24b0e6a040b7966225af26558ff8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } }
I20240501 14:05:33.916040 8227 raft_consensus.cc:729] T 9b88c43cb8704583a3e65c5c35c205f6 P 938a24b0e6a040b7966225af26558ff8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 938a24b0e6a040b7966225af26558ff8, State: Initialized, Role: LEARNER
I20240501 14:05:33.916249 8227 consensus_queue.cc:260] T 9b88c43cb8704583a3e65c5c35c205f6 P 938a24b0e6a040b7966225af26558ff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } }
I20240501 14:05:33.916787 8227 ts_tablet_manager.cc:1398] T 9b88c43cb8704583a3e65c5c35c205f6 P 938a24b0e6a040b7966225af26558ff8: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:05:33.917094 8205 tablet_bootstrap.cc:492] T f4d2b8c005b64bbb8e6621da068d15e5 P 19a6d2d8a08142d5bffcde3a37234692: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:05:33.917501 5406 tablet_copy_service.cc:342] P be56773105e744919601127a94e98adf: Request end of tablet copy session 938a24b0e6a040b7966225af26558ff8-9b88c43cb8704583a3e65c5c35c205f6 received from {username='slave'} at 127.29.124.133:50583
I20240501 14:05:33.917560 8205 tablet_bootstrap.cc:492] T f4d2b8c005b64bbb8e6621da068d15e5 P 19a6d2d8a08142d5bffcde3a37234692: Bootstrap complete.
I20240501 14:05:33.917641 5406 tablet_copy_service.cc:433] P be56773105e744919601127a94e98adf: ending tablet copy session 938a24b0e6a040b7966225af26558ff8-9b88c43cb8704583a3e65c5c35c205f6 on tablet 9b88c43cb8704583a3e65c5c35c205f6 with peer 938a24b0e6a040b7966225af26558ff8
I20240501 14:05:33.917810 8205 ts_tablet_manager.cc:1367] T f4d2b8c005b64bbb8e6621da068d15e5 P 19a6d2d8a08142d5bffcde3a37234692: Time spent bootstrapping tablet: real 0.314s user 0.041s sys 0.017s
I20240501 14:05:33.919179 8205 raft_consensus.cc:348] T f4d2b8c005b64bbb8e6621da068d15e5 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } }
I20240501 14:05:33.919430 8205 raft_consensus.cc:729] T f4d2b8c005b64bbb8e6621da068d15e5 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 19a6d2d8a08142d5bffcde3a37234692, State: Initialized, Role: LEARNER
I20240501 14:05:33.919631 8205 consensus_queue.cc:260] T f4d2b8c005b64bbb8e6621da068d15e5 P 19a6d2d8a08142d5bffcde3a37234692 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } }
I20240501 14:05:33.920315 8205 ts_tablet_manager.cc:1398] T f4d2b8c005b64bbb8e6621da068d15e5 P 19a6d2d8a08142d5bffcde3a37234692: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240501 14:05:33.922017 7155 raft_consensus.cc:2932] T 9fd94635797d447cbdc9181ee91f465b P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:33.922468 5126 raft_consensus.cc:2932] T 9fd94635797d447cbdc9181ee91f465b P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:33.923090 7156 raft_consensus.cc:2932] T ac4b234f794149e4b6def7a3b358942d P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:33.927048 4848 catalog_manager.cc:5484] T 9fd94635797d447cbdc9181ee91f465b P 4360b59e3a154688a79ca5f9c0a37bc9 reported cstate change: config changed from index 4 to 5, 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:33.930557 4846 catalog_manager.cc:5484] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 reported cstate change: config changed from index 4 to 5, 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:05:33.935655 4839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8cc5eec4600640f7a293877c825f8d96 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:05:33.935699 5384 consensus_queue.cc:237] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } }
I20240501 14:05:33.938175 7025 raft_consensus.cc:2932] T 8cc5eec4600640f7a293877c825f8d96 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:33.938988 5406 tablet_copy_service.cc:342] P be56773105e744919601127a94e98adf: Request end of tablet copy session 19a6d2d8a08142d5bffcde3a37234692-f4d2b8c005b64bbb8e6621da068d15e5 received from {username='slave'} at 127.29.124.134:56125
I20240501 14:05:33.939121 5406 tablet_copy_service.cc:433] P be56773105e744919601127a94e98adf: ending tablet copy session 19a6d2d8a08142d5bffcde3a37234692-f4d2b8c005b64bbb8e6621da068d15e5 on tablet f4d2b8c005b64bbb8e6621da068d15e5 with peer 19a6d2d8a08142d5bffcde3a37234692
I20240501 14:05:33.945540 5382 consensus_queue.cc:237] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } }
I20240501 14:05:33.946552 5385 raft_consensus.cc:2932] T ac4b234f794149e4b6def7a3b358942d P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:33.949945 5406 tablet_copy_service.cc:140] P be56773105e744919601127a94e98adf: Received BeginTabletCopySession request for tablet 9b88c43cb8704583a3e65c5c35c205f6 from peer 4360b59e3a154688a79ca5f9c0a37bc9 ({username='slave'} at 127.29.124.135:51309)
I20240501 14:05:33.950096 5406 tablet_copy_service.cc:161] P be56773105e744919601127a94e98adf: Beginning new tablet copy session on tablet 9b88c43cb8704583a3e65c5c35c205f6 from peer 4360b59e3a154688a79ca5f9c0a37bc9 at {username='slave'} at 127.29.124.135:51309: session id = 4360b59e3a154688a79ca5f9c0a37bc9-9b88c43cb8704583a3e65c5c35c205f6
I20240501 14:05:33.953126 5406 tablet_copy_source_session.cc:205] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:05:33.954828 8241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b88c43cb8704583a3e65c5c35c205f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:33.958755 5126 raft_consensus.cc:2932] T 8cc5eec4600640f7a293877c825f8d96 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:33.959764 8235 tablet_bootstrap.cc:492] T 9afe3b20a43d4b5bb800af39f945948d P 6dac90f01e744f9c8701a404440a4c77: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:05:33.960151 8235 tablet_bootstrap.cc:492] T 9afe3b20a43d4b5bb800af39f945948d P 6dac90f01e744f9c8701a404440a4c77: Bootstrap complete.
I20240501 14:05:33.960425 8235 ts_tablet_manager.cc:1367] T 9afe3b20a43d4b5bb800af39f945948d P 6dac90f01e744f9c8701a404440a4c77: Time spent bootstrapping tablet: real 0.382s user 0.048s sys 0.000s
I20240501 14:05:33.961207 8235 raft_consensus.cc:348] T 9afe3b20a43d4b5bb800af39f945948d P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } }
I20240501 14:05:33.961479 8235 raft_consensus.cc:729] T 9afe3b20a43d4b5bb800af39f945948d P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6dac90f01e744f9c8701a404440a4c77, State: Initialized, Role: LEARNER
I20240501 14:05:33.961675 8235 consensus_queue.cc:260] T 9afe3b20a43d4b5bb800af39f945948d P 6dac90f01e744f9c8701a404440a4c77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } }
I20240501 14:05:33.962780 8235 ts_tablet_manager.cc:1398] T 9afe3b20a43d4b5bb800af39f945948d P 6dac90f01e744f9c8701a404440a4c77: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20240501 14:05:33.963364 5146 tablet_copy_service.cc:342] P e9c60449cf3c46f8aca4ab6c535302a4: Request end of tablet copy session 6dac90f01e744f9c8701a404440a4c77-9afe3b20a43d4b5bb800af39f945948d received from {username='slave'} at 127.29.124.136:51401
I20240501 14:05:33.963476 5146 tablet_copy_service.cc:433] P e9c60449cf3c46f8aca4ab6c535302a4: ending tablet copy session 6dac90f01e744f9c8701a404440a4c77-9afe3b20a43d4b5bb800af39f945948d on tablet 9afe3b20a43d4b5bb800af39f945948d with peer 6dac90f01e744f9c8701a404440a4c77
I20240501 14:05:33.964861 6891 raft_consensus.cc:1203] T 9b88c43cb8704583a3e65c5c35c205f6 P 938a24b0e6a040b7966225af26558ff8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:05:33.965348 5126 consensus_queue.cc:237] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } }
I20240501 14:05:33.966730 8241 tablet_copy_client.cc:783] T 9b88c43cb8704583a3e65c5c35c205f6 P 4360b59e3a154688a79ca5f9c0a37bc9: tablet copy: Starting download of 0 data blocks...
I20240501 14:05:33.967020 8241 tablet_copy_client.cc:647] T 9b88c43cb8704583a3e65c5c35c205f6 P 4360b59e3a154688a79ca5f9c0a37bc9: tablet copy: Starting download of 1 WAL segments...
I20240501 14:05:33.967604 7156 raft_consensus.cc:1261] T ac4b234f794149e4b6def7a3b358942d P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:33.968981 8149 consensus_queue.cc:1035] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:33.970152 7286 raft_consensus.cc:1203] T 9afe3b20a43d4b5bb800af39f945948d P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:05:33.972692 5385 raft_consensus.cc:1261] T ac4b234f794149e4b6def7a3b358942d P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:33.973246 8175 raft_consensus.cc:2932] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:33.974551 7156 raft_consensus.cc:2932] T ac4b234f794149e4b6def7a3b358942d P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:33.976454 4991 consensus_queue.cc:237] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } }
I20240501 14:05:33.977102 8149 consensus_queue.cc:1035] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.004s
I20240501 14:05:33.977973 4993 raft_consensus.cc:2932] T ac4b234f794149e4b6def7a3b358942d P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
W20240501 14:05:33.979300 4931 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fee16905387d428aa02cd2070c86c790. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:33.979380 5126 raft_consensus.cc:1261] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:05:33.979946 8482 consensus_queue.cc:1035] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:05:33.980863 5383 raft_consensus.cc:2932] T ac4b234f794149e4b6def7a3b358942d P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:33.981753 4992 raft_consensus.cc:2932] T 9fd94635797d447cbdc9181ee91f465b P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:33.982482 4839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ac4b234f794149e4b6def7a3b358942d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:05:33.983400 7155 raft_consensus.cc:1261] T fee16905387d428aa02cd2070c86c790 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:05:33.983951 8222 consensus_queue.cc:1035] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:05:33.987403 8241 tablet_copy_client.cc:523] T 9b88c43cb8704583a3e65c5c35c205f6 P 4360b59e3a154688a79ca5f9c0a37bc9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:05:33.987615 4851 catalog_manager.cc:5484] T ac4b234f794149e4b6def7a3b358942d P 4360b59e3a154688a79ca5f9c0a37bc9 reported cstate change: config changed from index 5 to 6, VOTER 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129) evicted. New cstate: current_term: 1 leader_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:33.989712 4839 catalog_manager.cc:4830] TS 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129:44033): tablet 9d32e0707d244f2c9bbcb57f62b09f31 (table rebalance_test_table_8 [id=f26ba2f4e5654a5fa72021df5929200d]) successfully deleted
I20240501 14:05:33.990442 5381 raft_consensus.cc:1261] T fee16905387d428aa02cd2070c86c790 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240501 14:05:33.992170 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2c39b53867e047faaf24cc6f87a8100c. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:33.992183 4973 tablet_service.cc:1514] Processing DeleteTablet for tablet ac4b234f794149e4b6def7a3b358942d with delete_type TABLET_DATA_TOMBSTONED (TS 04088ee3bea440a7bd5ff772152b9410 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41194
I20240501 14:05:33.992617 8241 tablet_bootstrap.cc:492] T 9b88c43cb8704583a3e65c5c35c205f6 P 4360b59e3a154688a79ca5f9c0a37bc9: Bootstrap starting.
I20240501 14:05:33.993204 6891 raft_consensus.cc:1261] T 2c39b53867e047faaf24cc6f87a8100c P 938a24b0e6a040b7966225af26558ff8 [term 1 LEARNER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:33.993861 8352 consensus_queue.cc:1035] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:33.997249 5255 consensus_queue.cc:237] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } }
I20240501 14:05:33.999125 8363 consensus_queue.cc:1035] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:05:33.999883 8375 raft_consensus.cc:2932] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:34.000947 7155 raft_consensus.cc:2932] T fee16905387d428aa02cd2070c86c790 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:34.002259 8285 tablet_copy_client.cc:308] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 19a6d2d8a08142d5bffcde3a37234692: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.131:33209
I20240501 14:05:34.004361 4839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fee16905387d428aa02cd2070c86c790 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:05:34.004957 5276 tablet_copy_service.cc:140] P 1c3fdc5272484ac09ef3e5e36a08f27c: Received BeginTabletCopySession request for tablet 23ca2bf1e59f44fcac8a48aff8d056b1 from peer 19a6d2d8a08142d5bffcde3a37234692 ({username='slave'} at 127.29.124.134:43435)
I20240501 14:05:34.005136 5276 tablet_copy_service.cc:161] P 1c3fdc5272484ac09ef3e5e36a08f27c: Beginning new tablet copy session on tablet 23ca2bf1e59f44fcac8a48aff8d056b1 from peer 19a6d2d8a08142d5bffcde3a37234692 at {username='slave'} at 127.29.124.134:43435: session id = 19a6d2d8a08142d5bffcde3a37234692-23ca2bf1e59f44fcac8a48aff8d056b1
I20240501 14:05:34.005815 5126 raft_consensus.cc:2932] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:34.007966 5276 tablet_copy_source_session.cc:205] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:05:34.009363 8285 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23ca2bf1e59f44fcac8a48aff8d056b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:34.010025 7156 raft_consensus.cc:1261] T 9fd94635797d447cbdc9181ee91f465b P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:34.010531 7025 raft_consensus.cc:1203] T 1a7bbfc15a76450bbc9197037e856e08 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:05:34.011669 7689 consensus_queue.cc:1035] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:34.011893 4992 raft_consensus.cc:1261] T 9fd94635797d447cbdc9181ee91f465b P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:34.012409 7689 consensus_queue.cc:1035] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:34.013443 5385 raft_consensus.cc:2932] T fee16905387d428aa02cd2070c86c790 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:34.013854 4993 consensus_queue.cc:237] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } }
I20240501 14:05:34.016306 7689 raft_consensus.cc:2932] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:34.017405 5125 raft_consensus.cc:1261] T 8cc5eec4600640f7a293877c825f8d96 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:05:34.017402 7025 raft_consensus.cc:1261] T 8cc5eec4600640f7a293877c825f8d96 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:05:34.017920 8222 consensus_queue.cc:1035] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:05:34.018941 8222 consensus_queue.cc:1035] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:05:34.020282 4838 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9fd94635797d447cbdc9181ee91f465b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:05:34.021745 5254 raft_consensus.cc:1261] T 2c39b53867e047faaf24cc6f87a8100c P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:34.022485 8467 raft_consensus.cc:2932] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER be56773105e744919601127a94e98adf (127.29.124.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:34.020990 4848 catalog_manager.cc:5484] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 reported cstate change: config changed from index 7 to 8, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) evicted. New cstate: current_term: 1 leader_uuid: "04088ee3bea440a7bd5ff772152b9410" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:34.024274 4993 raft_consensus.cc:2932] T 9fd94635797d447cbdc9181ee91f465b P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:34.025980 7156 raft_consensus.cc:2932] T 9fd94635797d447cbdc9181ee91f465b P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:34.027235 4847 catalog_manager.cc:5484] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c reported cstate change: config changed from index 5 to 6, VOTER e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130) evicted. New cstate: current_term: 1 leader_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:05:34.027331 5125 raft_consensus.cc:2932] T 8cc5eec4600640f7a293877c825f8d96 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER be56773105e744919601127a94e98adf (127.29.124.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:34.031710 4995 consensus_queue.cc:237] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } }
I20240501 14:05:34.032895 7025 raft_consensus.cc:2932] T 8cc5eec4600640f7a293877c825f8d96 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER be56773105e744919601127a94e98adf (127.29.124.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:34.033443 4854 catalog_manager.cc:5484] T 8cc5eec4600640f7a293877c825f8d96 P e9c60449cf3c46f8aca4ab6c535302a4 reported cstate change: config changed from index 7 to 8, VOTER be56773105e744919601127a94e98adf (127.29.124.132) evicted. New cstate: current_term: 1 leader_uuid: "04088ee3bea440a7bd5ff772152b9410" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:34.037384 5365 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cc5eec4600640f7a293877c825f8d96 with delete_type TABLET_DATA_TOMBSTONED (TS be56773105e744919601127a94e98adf not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:43020
I20240501 14:05:34.037883 8175 raft_consensus.cc:1050] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4: attempting to promote NON_VOTER 6dac90f01e744f9c8701a404440a4c77 to VOTER
I20240501 14:05:34.038637 8175 consensus_queue.cc:237] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } }
I20240501 14:05:34.040089 4839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8cc5eec4600640f7a293877c825f8d96 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:05:34.040524 7156 raft_consensus.cc:1261] T fee16905387d428aa02cd2070c86c790 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:05:34.041363 5125 raft_consensus.cc:1261] T 2c39b53867e047faaf24cc6f87a8100c P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:34.041651 8222 consensus_queue.cc:1035] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:05:34.043023 8386 tablet_replica.cc:330] T ac4b234f794149e4b6def7a3b358942d P 04088ee3bea440a7bd5ff772152b9410: stopping tablet replica
I20240501 14:05:34.043272 8386 raft_consensus.cc:2229] T ac4b234f794149e4b6def7a3b358942d P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:05:34.043785 8386 raft_consensus.cc:2258] T ac4b234f794149e4b6def7a3b358942d P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:05:34.046372 8352 consensus_queue.cc:1035] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:34.049664 5125 raft_consensus.cc:1261] T 61a274b24f53458f88c2cde4f8dbe4f4 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:05:34.051167 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 61a274b24f53458f88c2cde4f8dbe4f4. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:34.055914 6872 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cc5eec4600640f7a293877c825f8d96 with delete_type TABLET_DATA_TOMBSTONED (TS 938a24b0e6a040b7966225af26558ff8 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:50548
I20240501 14:05:34.056568 6871 tablet_service.cc:1514] Processing DeleteTablet for tablet fee16905387d428aa02cd2070c86c790 with delete_type TABLET_DATA_TOMBSTONED (TS 938a24b0e6a040b7966225af26558ff8 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:50548
I20240501 14:05:34.057633 5106 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fd94635797d447cbdc9181ee91f465b with delete_type TABLET_DATA_TOMBSTONED (TS e9c60449cf3c46f8aca4ab6c535302a4 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41850
I20240501 14:05:34.059860 8386 ts_tablet_manager.cc:1875] T ac4b234f794149e4b6def7a3b358942d P 04088ee3bea440a7bd5ff772152b9410: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:05:34.061424 8352 consensus_queue.cc:1035] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:05:34.065690 8352 raft_consensus.cc:2932] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } }
I20240501 14:05:34.071920 4847 catalog_manager.cc:5484] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf reported cstate change: config changed from index 4 to 5, NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) added. New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:05:34.073132 5126 raft_consensus.cc:1261] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:05:34.075309 8409 tablet_copy_client.cc:308] T 3d6859d53fd747a5a5e94722626fa522 P 938a24b0e6a040b7966225af26558ff8: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.130:34377
W20240501 14:05:34.076701 4840 catalog_manager.cc:4797] TS 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): delete failed for tablet 8cc5eec4600640f7a293877c825f8d96 because the tablet was not found. No further retry: Not found: Tablet not found: 8cc5eec4600640f7a293877c825f8d96
I20240501 14:05:34.077441 8505 tablet_replica.cc:330] T fee16905387d428aa02cd2070c86c790 P 938a24b0e6a040b7966225af26558ff8: stopping tablet replica
I20240501 14:05:34.078049 8505 raft_consensus.cc:2229] T fee16905387d428aa02cd2070c86c790 P 938a24b0e6a040b7966225af26558ff8 [term 1 LEARNER]: Raft consensus shutting down.
I20240501 14:05:34.078485 8505 raft_consensus.cc:2258] T fee16905387d428aa02cd2070c86c790 P 938a24b0e6a040b7966225af26558ff8 [term 1 LEARNER]: Raft consensus is shut down!
I20240501 14:05:34.080457 8505 ts_tablet_manager.cc:1875] T fee16905387d428aa02cd2070c86c790 P 938a24b0e6a040b7966225af26558ff8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:05:34.083159 8506 tablet_replica.cc:330] T 9fd94635797d447cbdc9181ee91f465b P e9c60449cf3c46f8aca4ab6c535302a4: stopping tablet replica
I20240501 14:05:34.083652 8506 raft_consensus.cc:2229] T 9fd94635797d447cbdc9181ee91f465b P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:05:34.084043 8506 raft_consensus.cc:2258] T 9fd94635797d447cbdc9181ee91f465b P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240501 14:05:34.087086 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e7d5f02ccba34254a4d20951878b1214. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:34.088276 8222 consensus_queue.cc:1035] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.015s
I20240501 14:05:34.089116 8506 ts_tablet_manager.cc:1875] T 9fd94635797d447cbdc9181ee91f465b P e9c60449cf3c46f8aca4ab6c535302a4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:05:34.091224 8247 raft_consensus.cc:1050] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf: attempting to promote NON_VOTER 938a24b0e6a040b7966225af26558ff8 to VOTER
I20240501 14:05:34.092135 8247 consensus_queue.cc:237] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } }
I20240501 14:05:34.096760 8386 ts_tablet_manager.cc:1888] T ac4b234f794149e4b6def7a3b358942d P 04088ee3bea440a7bd5ff772152b9410: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:05:34.096892 8386 log.cc:1196] T ac4b234f794149e4b6def7a3b358942d P 04088ee3bea440a7bd5ff772152b9410: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/wal/wals/ac4b234f794149e4b6def7a3b358942d
I20240501 14:05:34.101845 6891 raft_consensus.cc:1261] T 2c39b53867e047faaf24cc6f87a8100c P 938a24b0e6a040b7966225af26558ff8 [term 1 LEARNER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:34.102284 7024 raft_consensus.cc:1203] T f4d2b8c005b64bbb8e6621da068d15e5 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240501 14:05:34.104413 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2c39b53867e047faaf24cc6f87a8100c. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:34.105844 8512 consensus_queue.cc:1035] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:34.107246 6891 raft_consensus.cc:2932] T 2c39b53867e047faaf24cc6f87a8100c P 938a24b0e6a040b7966225af26558ff8 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } }
I20240501 14:05:34.110406 5385 raft_consensus.cc:1261] T fee16905387d428aa02cd2070c86c790 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:05:34.112115 5126 raft_consensus.cc:1261] T 2c39b53867e047faaf24cc6f87a8100c P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:34.112782 8506 ts_tablet_manager.cc:1888] T 9fd94635797d447cbdc9181ee91f465b P e9c60449cf3c46f8aca4ab6c535302a4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:05:34.112931 8506 log.cc:1196] T 9fd94635797d447cbdc9181ee91f465b P e9c60449cf3c46f8aca4ab6c535302a4: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/wal/wals/9fd94635797d447cbdc9181ee91f465b
I20240501 14:05:34.112795 8222 consensus_queue.cc:1035] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:05:34.113813 4995 raft_consensus.cc:1261] T e7d5f02ccba34254a4d20951878b1214 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:05:34.117090 8512 consensus_queue.cc:1035] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:34.117324 8505 ts_tablet_manager.cc:1888] T fee16905387d428aa02cd2070c86c790 P 938a24b0e6a040b7966225af26558ff8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 14:05:34.117481 8505 log.cc:1196] T fee16905387d428aa02cd2070c86c790 P 938a24b0e6a040b7966225af26558ff8: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/wal/wals/fee16905387d428aa02cd2070c86c790
I20240501 14:05:34.117338 8352 raft_consensus.cc:2932] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } }
I20240501 14:05:34.118224 4840 catalog_manager.cc:4830] TS 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): tablet fee16905387d428aa02cd2070c86c790 (table rebalance_test_table_2 [id=facc024bfd644356a8879fd9661c928d]) successfully deleted
I20240501 14:05:34.117957 5125 raft_consensus.cc:2932] T 2c39b53867e047faaf24cc6f87a8100c P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } }
I20240501 14:05:34.119333 4839 catalog_manager.cc:4830] TS e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377): tablet 9fd94635797d447cbdc9181ee91f465b (table rebalance_test_table_5 [id=3794b1ccb5004f7cb92a229ef6ce3595]) successfully deleted
I20240501 14:05:34.120532 8247 consensus_queue.cc:1035] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:05:34.121038 5126 raft_consensus.cc:2932] T 61a274b24f53458f88c2cde4f8dbe4f4 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } }
I20240501 14:05:34.122884 4853 catalog_manager.cc:5484] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf reported cstate change: config changed from index -1 to 4, NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) added. New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:05:34.124531 4839 catalog_manager.cc:4830] TS 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129:44033): tablet ac4b234f794149e4b6def7a3b358942d (table rebalance_test_table_6 [id=3c1513e944ee4b518373c4efa5e200cc]) successfully deleted
I20240501 14:05:34.133810 8375 raft_consensus.cc:2932] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:34.136904 8501 tablet_replica.cc:330] T 8cc5eec4600640f7a293877c825f8d96 P be56773105e744919601127a94e98adf: stopping tablet replica
I20240501 14:05:34.137689 8501 raft_consensus.cc:2229] T 8cc5eec4600640f7a293877c825f8d96 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:05:34.138242 8501 raft_consensus.cc:2258] T 8cc5eec4600640f7a293877c825f8d96 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:05:34.139510 4839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fee16905387d428aa02cd2070c86c790 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:05:34.140843 5255 raft_consensus.cc:1261] T 61a274b24f53458f88c2cde4f8dbe4f4 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:05:34.140694 7156 raft_consensus.cc:2932] T fee16905387d428aa02cd2070c86c790 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:34.141160 8501 ts_tablet_manager.cc:1875] T 8cc5eec4600640f7a293877c825f8d96 P be56773105e744919601127a94e98adf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:05:34.141640 5255 raft_consensus.cc:1261] T e7d5f02ccba34254a4d20951878b1214 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:05:34.142153 5255 raft_consensus.cc:1261] T 2c39b53867e047faaf24cc6f87a8100c P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:34.141978 4853 catalog_manager.cc:5484] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 reported cstate change: config changed from index 8 to 9, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) evicted. New cstate: current_term: 1 leader_uuid: "04088ee3bea440a7bd5ff772152b9410" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:05:34.143863 8247 consensus_queue.cc:1035] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:05:34.144402 8247 consensus_queue.cc:1035] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:05:34.144837 8247 consensus_queue.cc:1035] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:34.145648 7286 raft_consensus.cc:1261] T f9cb784f60774ea7b6e70ccfebf867a8 P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:34.147403 8273 consensus_queue.cc:1035] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:34.153923 5385 consensus_queue.cc:237] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } }
I20240501 14:05:34.157763 7267 tablet_service.cc:1514] Processing DeleteTablet for tablet fee16905387d428aa02cd2070c86c790 with delete_type TABLET_DATA_TOMBSTONED (TS 6dac90f01e744f9c8701a404440a4c77 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:52244
I20240501 14:05:34.165988 5382 raft_consensus.cc:1261] T f9cb784f60774ea7b6e70ccfebf867a8 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:34.166548 8279 consensus_queue.cc:1035] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240501 14:05:34.175653 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 61a274b24f53458f88c2cde4f8dbe4f4. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:34.178478 5125 raft_consensus.cc:1261] T 61a274b24f53458f88c2cde4f8dbe4f4 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:34.180428 8352 raft_consensus.cc:2932] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } }
I20240501 14:05:34.185782 6891 raft_consensus.cc:2932] T 2c39b53867e047faaf24cc6f87a8100c P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } }
I20240501 14:05:34.185909 5124 raft_consensus.cc:2932] T 2c39b53867e047faaf24cc6f87a8100c P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } }
W20240501 14:05:34.188664 4841 catalog_manager.cc:4797] TS 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): delete failed for tablet fee16905387d428aa02cd2070c86c790 because the tablet was not found. No further retry: Not found: Tablet not found: fee16905387d428aa02cd2070c86c790
I20240501 14:05:34.191315 4847 catalog_manager.cc:5484] T 2c39b53867e047faaf24cc6f87a8100c P 938a24b0e6a040b7966225af26558ff8 reported cstate change: config changed from index 5 to 6, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } }
I20240501 14:05:34.191948 4995 consensus_queue.cc:237] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } }
I20240501 14:05:34.194195 7156 raft_consensus.cc:1261] T fee16905387d428aa02cd2070c86c790 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:05:34.194295 5126 raft_consensus.cc:2932] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:34.194936 8375 consensus_queue.cc:1035] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:05:34.197463 8362 raft_consensus.cc:2932] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER be56773105e744919601127a94e98adf (127.29.124.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:34.201220 4839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fee16905387d428aa02cd2070c86c790 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:05:34.201942 8503 consensus_queue.cc:1035] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.016s
I20240501 14:05:34.202769 7156 raft_consensus.cc:2932] T fee16905387d428aa02cd2070c86c790 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER be56773105e744919601127a94e98adf (127.29.124.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:34.206697 8227 ts_tablet_manager.cc:902] T 71660575d94748d8a9329f909a42a0cd P 938a24b0e6a040b7966225af26558ff8: Initiating tablet copy from peer 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209)
I20240501 14:05:34.207969 8501 ts_tablet_manager.cc:1888] T 8cc5eec4600640f7a293877c825f8d96 P be56773105e744919601127a94e98adf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240501 14:05:34.208146 8501 log.cc:1196] T 8cc5eec4600640f7a293877c825f8d96 P be56773105e744919601127a94e98adf: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/wal/wals/8cc5eec4600640f7a293877c825f8d96
I20240501 14:05:34.209225 5256 raft_consensus.cc:858] T 2c39b53867e047faaf24cc6f87a8100c P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }.
Pending config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } }
W20240501 14:05:34.209359 5256 raft_consensus.cc:1565] T 2c39b53867e047faaf24cc6f87a8100c P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }.
Pending config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } }
I20240501 14:05:34.209167 5385 raft_consensus.cc:2932] T fee16905387d428aa02cd2070c86c790 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:34.211441 4846 catalog_manager.cc:5484] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 reported cstate change: config changed from index 9 to 10, VOTER be56773105e744919601127a94e98adf (127.29.124.132) evicted. New cstate: current_term: 1 leader_uuid: "04088ee3bea440a7bd5ff772152b9410" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:05:34.220245 8513 raft_consensus.cc:2932] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:34.224483 5256 raft_consensus.cc:2932] T 2c39b53867e047faaf24cc6f87a8100c P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } }
I20240501 14:05:34.227528 5124 raft_consensus.cc:2932] T 61a274b24f53458f88c2cde4f8dbe4f4 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:34.230515 5125 raft_consensus.cc:1261] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:05:34.230434 4846 catalog_manager.cc:5484] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf reported cstate change: config changed from index 4 to 5, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:05:34.231267 8375 consensus_queue.cc:1035] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:05:34.233183 5125 raft_consensus.cc:2932] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER be56773105e744919601127a94e98adf (127.29.124.132) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:34.236964 4839 catalog_manager.cc:4830] TS be56773105e744919601127a94e98adf (127.29.124.132:35459): tablet 8cc5eec4600640f7a293877c825f8d96 (table rebalance_test_table_6 [id=3c1513e944ee4b518373c4efa5e200cc]) successfully deleted
I20240501 14:05:34.237541 8352 raft_consensus.cc:2932] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } }
I20240501 14:05:34.244858 4992 raft_consensus.cc:2932] T e7d5f02ccba34254a4d20951878b1214 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } }
I20240501 14:05:34.247673 4854 catalog_manager.cc:5484] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf reported cstate change: config changed from index -1 to 4, NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) added. New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:05:34.248831 7285 raft_consensus.cc:2932] T f9cb784f60774ea7b6e70ccfebf867a8 P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:34.255456 5365 tablet_service.cc:1514] Processing DeleteTablet for tablet fee16905387d428aa02cd2070c86c790 with delete_type TABLET_DATA_TOMBSTONED (TS be56773105e744919601127a94e98adf not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:43020
I20240501 14:05:34.255947 8501 tablet_replica.cc:330] T fee16905387d428aa02cd2070c86c790 P be56773105e744919601127a94e98adf: stopping tablet replica
I20240501 14:05:34.256140 8501 raft_consensus.cc:2229] T fee16905387d428aa02cd2070c86c790 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:05:34.247834 8279 raft_consensus.cc:2932] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:34.256565 8501 raft_consensus.cc:2258] T fee16905387d428aa02cd2070c86c790 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:05:34.256515 4846 catalog_manager.cc:5484] T f9cb784f60774ea7b6e70ccfebf867a8 P 6dac90f01e744f9c8701a404440a4c77 reported cstate change: config changed from index 4 to 5, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:34.257803 8501 ts_tablet_manager.cc:1875] T fee16905387d428aa02cd2070c86c790 P be56773105e744919601127a94e98adf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:05:34.258788 5385 consensus_queue.cc:237] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:05:34.261087 6891 raft_consensus.cc:1261] T 2c39b53867e047faaf24cc6f87a8100c P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:05:34.261386 5125 raft_consensus.cc:1261] T 2c39b53867e047faaf24cc6f87a8100c P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:05:34.261839 8513 consensus_queue.cc:1035] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:05:34.262157 8503 consensus_queue.cc:1035] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:05:34.261847 5382 raft_consensus.cc:2932] T f9cb784f60774ea7b6e70ccfebf867a8 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:34.269717 8501 ts_tablet_manager.cc:1888] T fee16905387d428aa02cd2070c86c790 P be56773105e744919601127a94e98adf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240501 14:05:34.269948 8501 log.cc:1196] T fee16905387d428aa02cd2070c86c790 P be56773105e744919601127a94e98adf: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/wal/wals/fee16905387d428aa02cd2070c86c790
I20240501 14:05:34.270743 4839 catalog_manager.cc:4830] TS be56773105e744919601127a94e98adf (127.29.124.132:35459): tablet fee16905387d428aa02cd2070c86c790 (table rebalance_test_table_2 [id=facc024bfd644356a8879fd9661c928d]) successfully deleted
W20240501 14:05:34.271132 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 61a274b24f53458f88c2cde4f8dbe4f4. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:34.272161 8503 raft_consensus.cc:2932] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:34.276454 4839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2c39b53867e047faaf24cc6f87a8100c with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:05:34.277592 5124 consensus_queue.cc:237] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } }
I20240501 14:05:34.277683 5126 raft_consensus.cc:2932] T 2c39b53867e047faaf24cc6f87a8100c P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:34.277180 6891 raft_consensus.cc:2932] T 2c39b53867e047faaf24cc6f87a8100c P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:34.280143 7285 raft_consensus.cc:1261] T f9cb784f60774ea7b6e70ccfebf867a8 P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:34.280721 8273 consensus_queue.cc:1035] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:34.282238 4847 catalog_manager.cc:5484] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf reported cstate change: config changed from index 6 to 7, NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) evicted. New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:05:34.285588 5382 raft_consensus.cc:1261] T f9cb784f60774ea7b6e70ccfebf867a8 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:34.286166 8273 consensus_queue.cc:1035] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:34.294637 5382 consensus_queue.cc:237] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:05:34.297817 6891 raft_consensus.cc:1261] T 2c39b53867e047faaf24cc6f87a8100c P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:05:34.298430 8503 consensus_queue.cc:1035] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:05:34.300155 8273 raft_consensus.cc:2932] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:34.300696 8352 raft_consensus.cc:2932] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:34.303501 4839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2c39b53867e047faaf24cc6f87a8100c with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:05:34.303948 6892 raft_consensus.cc:2932] T 2c39b53867e047faaf24cc6f87a8100c P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:34.304665 8227 tablet_copy_client.cc:308] T 71660575d94748d8a9329f909a42a0cd P 938a24b0e6a040b7966225af26558ff8: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.131:33209
I20240501 14:05:34.310029 4839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f9cb784f60774ea7b6e70ccfebf867a8 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:05:34.312176 5125 raft_consensus.cc:1261] T 2c39b53867e047faaf24cc6f87a8100c P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:05:34.312816 8352 consensus_queue.cc:1035] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:05:34.313274 5385 raft_consensus.cc:2932] T f9cb784f60774ea7b6e70ccfebf867a8 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:34.314600 5125 raft_consensus.cc:2932] T 2c39b53867e047faaf24cc6f87a8100c P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:34.314983 4847 catalog_manager.cc:5484] T 2c39b53867e047faaf24cc6f87a8100c P 938a24b0e6a040b7966225af26558ff8 reported cstate change: config changed from index 7 to 8, VOTER 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) evicted. New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:34.314951 7287 raft_consensus.cc:2932] T f9cb784f60774ea7b6e70ccfebf867a8 P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:34.320070 4851 catalog_manager.cc:5484] T f9cb784f60774ea7b6e70ccfebf867a8 P 6dac90f01e744f9c8701a404440a4c77 reported cstate change: config changed from index 5 to 6, VOTER 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) evicted. New cstate: current_term: 1 leader_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:34.324894 5251 raft_consensus.cc:1261] T 2c39b53867e047faaf24cc6f87a8100c P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:05:34.325587 5251 raft_consensus.cc:1261] T f9cb784f60774ea7b6e70ccfebf867a8 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:34.326481 5236 tablet_service.cc:1514] Processing DeleteTablet for tablet f9cb784f60774ea7b6e70ccfebf867a8 with delete_type TABLET_DATA_TOMBSTONED (TS 1c3fdc5272484ac09ef3e5e36a08f27c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39020
I20240501 14:05:34.328802 5251 consensus_queue.cc:237] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } }
I20240501 14:05:34.329061 5234 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c39b53867e047faaf24cc6f87a8100c with delete_type TABLET_DATA_TOMBSTONED (TS 1c3fdc5272484ac09ef3e5e36a08f27c not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:39020
I20240501 14:05:34.333214 4992 raft_consensus.cc:1261] T 51cbcf4888c440c2ad096c988ab2d968 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:05:34.333819 8500 consensus_queue.cc:1035] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:05:34.338724 5385 raft_consensus.cc:1261] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:05:34.341339 8500 consensus_queue.cc:1035] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:05:34.351440 5255 raft_consensus.cc:2932] T 61a274b24f53458f88c2cde4f8dbe4f4 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } }
I20240501 14:05:34.366809 4992 raft_consensus.cc:2932] T 51cbcf4888c440c2ad096c988ab2d968 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:34.365340 8500 raft_consensus.cc:2932] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:34.372328 4846 catalog_manager.cc:5484] T 51cbcf4888c440c2ad096c988ab2d968 P 04088ee3bea440a7bd5ff772152b9410 reported cstate change: config changed from index -1 to 4, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) added. New cstate: current_term: 1 leader_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:34.376313 5146 tablet_copy_service.cc:140] P e9c60449cf3c46f8aca4ab6c535302a4: Received BeginTabletCopySession request for tablet 3d6859d53fd747a5a5e94722626fa522 from peer 938a24b0e6a040b7966225af26558ff8 ({username='slave'} at 127.29.124.133:48619)
I20240501 14:05:34.376472 5146 tablet_copy_service.cc:161] P e9c60449cf3c46f8aca4ab6c535302a4: Beginning new tablet copy session on tablet 3d6859d53fd747a5a5e94722626fa522 from peer 938a24b0e6a040b7966225af26558ff8 at {username='slave'} at 127.29.124.133:48619: session id = 938a24b0e6a040b7966225af26558ff8-3d6859d53fd747a5a5e94722626fa522
I20240501 14:05:34.379379 5146 tablet_copy_source_session.cc:205] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:05:34.380821 8409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d6859d53fd747a5a5e94722626fa522. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:34.381880 8557 ts_tablet_manager.cc:902] T ba6cba753e934e719c730593c0706346 P 4360b59e3a154688a79ca5f9c0a37bc9: Initiating tablet copy from peer 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129:44033)
I20240501 14:05:34.387521 8409 tablet_copy_client.cc:783] T 3d6859d53fd747a5a5e94722626fa522 P 938a24b0e6a040b7966225af26558ff8: tablet copy: Starting download of 0 data blocks...
I20240501 14:05:34.387842 8409 tablet_copy_client.cc:647] T 3d6859d53fd747a5a5e94722626fa522 P 938a24b0e6a040b7966225af26558ff8: tablet copy: Starting download of 1 WAL segments...
I20240501 14:05:34.389737 8409 tablet_copy_client.cc:523] T 3d6859d53fd747a5a5e94722626fa522 P 938a24b0e6a040b7966225af26558ff8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:05:34.390405 5385 raft_consensus.cc:2932] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:34.394284 8409 tablet_bootstrap.cc:492] T 3d6859d53fd747a5a5e94722626fa522 P 938a24b0e6a040b7966225af26558ff8: Bootstrap starting.
I20240501 14:05:34.395319 5251 consensus_queue.cc:237] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }
I20240501 14:05:34.398345 4992 raft_consensus.cc:1261] T 51cbcf4888c440c2ad096c988ab2d968 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:34.398885 8281 consensus_queue.cc:1035] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:34.401610 8488 raft_consensus.cc:2932] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:34.405539 4992 raft_consensus.cc:2932] T 51cbcf4888c440c2ad096c988ab2d968 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:34.411376 5385 raft_consensus.cc:1261] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:34.411933 8500 consensus_queue.cc:1035] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:34.413606 5382 raft_consensus.cc:2932] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:34.416661 4846 catalog_manager.cc:5484] T 51cbcf4888c440c2ad096c988ab2d968 P 04088ee3bea440a7bd5ff772152b9410 reported cstate change: config changed from index 4 to 5, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New cstate: current_term: 1 leader_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:34.444247 8383 raft_consensus.cc:1050] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4: attempting to promote NON_VOTER 6dac90f01e744f9c8701a404440a4c77 to VOTER
I20240501 14:05:34.445149 8383 consensus_queue.cc:237] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } }
I20240501 14:05:34.460413 4992 raft_consensus.cc:1261] T 9afe3b20a43d4b5bb800af39f945948d P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:34.461166 8273 consensus_queue.cc:1035] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:34.466495 7286 raft_consensus.cc:1261] T 9afe3b20a43d4b5bb800af39f945948d P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:34.467511 8273 consensus_queue.cc:1035] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:34.478474 5251 raft_consensus.cc:2932] T e7d5f02ccba34254a4d20951878b1214 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } }
I20240501 14:05:34.484593 5254 consensus_queue.cc:237] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } }
I20240501 14:05:34.487778 5382 raft_consensus.cc:1261] T 71660575d94748d8a9329f909a42a0cd P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:34.488950 8277 consensus_queue.cc:1035] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:34.489800 4995 raft_consensus.cc:1261] T 71660575d94748d8a9329f909a42a0cd P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:34.490711 8277 consensus_queue.cc:1035] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:34.494240 8277 raft_consensus.cc:2932] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:34.498926 5382 raft_consensus.cc:2932] T 71660575d94748d8a9329f909a42a0cd P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:34.503547 4995 raft_consensus.cc:2932] T 71660575d94748d8a9329f909a42a0cd P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:34.512923 4846 catalog_manager.cc:5484] T 71660575d94748d8a9329f909a42a0cd P 04088ee3bea440a7bd5ff772152b9410 reported cstate change: config changed from index 4 to 5, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New cstate: current_term: 1 leader_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:34.517433 8549 tablet_replica.cc:330] T f9cb784f60774ea7b6e70ccfebf867a8 P 1c3fdc5272484ac09ef3e5e36a08f27c: stopping tablet replica
I20240501 14:05:34.521467 8383 raft_consensus.cc:2932] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:34.526604 7287 raft_consensus.cc:2932] T 9afe3b20a43d4b5bb800af39f945948d P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:34.529942 8503 raft_consensus.cc:1050] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf: attempting to promote NON_VOTER 938a24b0e6a040b7966225af26558ff8 to VOTER
I20240501 14:05:34.530915 8503 consensus_queue.cc:237] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } }
I20240501 14:05:34.531641 4992 raft_consensus.cc:2932] T 9afe3b20a43d4b5bb800af39f945948d P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:34.535619 4846 catalog_manager.cc:5484] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 reported cstate change: config changed from index 4 to 5, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:05:34.538362 8241 tablet_bootstrap.cc:492] T 9b88c43cb8704583a3e65c5c35c205f6 P 4360b59e3a154688a79ca5f9c0a37bc9: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:05:34.538748 8241 tablet_bootstrap.cc:492] T 9b88c43cb8704583a3e65c5c35c205f6 P 4360b59e3a154688a79ca5f9c0a37bc9: Bootstrap complete.
I20240501 14:05:34.539036 8241 ts_tablet_manager.cc:1367] T 9b88c43cb8704583a3e65c5c35c205f6 P 4360b59e3a154688a79ca5f9c0a37bc9: Time spent bootstrapping tablet: real 0.547s user 0.091s sys 0.063s
I20240501 14:05:34.539871 8241 raft_consensus.cc:348] T 9b88c43cb8704583a3e65c5c35c205f6 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } }
I20240501 14:05:34.540135 8241 raft_consensus.cc:729] T 9b88c43cb8704583a3e65c5c35c205f6 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4360b59e3a154688a79ca5f9c0a37bc9, State: Initialized, Role: LEARNER
I20240501 14:05:34.540328 8241 consensus_queue.cc:260] T 9b88c43cb8704583a3e65c5c35c205f6 P 4360b59e3a154688a79ca5f9c0a37bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } }
I20240501 14:05:34.540433 5124 consensus_queue.cc:237] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } }
W20240501 14:05:34.547410 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:34.548619 6892 raft_consensus.cc:1261] T 9b88c43cb8704583a3e65c5c35c205f6 P 938a24b0e6a040b7966225af26558ff8 [term 1 LEARNER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:34.548717 5251 raft_consensus.cc:1261] T 9afe3b20a43d4b5bb800af39f945948d P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:34.549337 8352 consensus_queue.cc:1035] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 14:05:34.549741 5192 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 71660575d94748d8a9329f909a42a0cd. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:34.552174 8549 raft_consensus.cc:2229] T f9cb784f60774ea7b6e70ccfebf867a8 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:05:34.555692 4993 raft_consensus.cc:1261] T 9b88c43cb8704583a3e65c5c35c205f6 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:34.556290 8503 consensus_queue.cc:1035] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:34.556440 5256 raft_consensus.cc:1261] T 9b88c43cb8704583a3e65c5c35c205f6 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:34.559396 8549 raft_consensus.cc:2258] T f9cb784f60774ea7b6e70ccfebf867a8 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:05:34.561303 7286 raft_consensus.cc:1261] T 9afe3b20a43d4b5bb800af39f945948d P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:34.562217 8273 consensus_queue.cc:1035] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:34.565300 8273 raft_consensus.cc:2932] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:34.567633 8241 ts_tablet_manager.cc:1398] T 9b88c43cb8704583a3e65c5c35c205f6 P 4360b59e3a154688a79ca5f9c0a37bc9: Time spent starting tablet: real 0.028s user 0.003s sys 0.000s
I20240501 14:05:34.568037 7286 raft_consensus.cc:2932] T 9afe3b20a43d4b5bb800af39f945948d P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:34.571710 5406 tablet_copy_service.cc:342] P be56773105e744919601127a94e98adf: Request end of tablet copy session 4360b59e3a154688a79ca5f9c0a37bc9-9b88c43cb8704583a3e65c5c35c205f6 received from {username='slave'} at 127.29.124.135:51309
I20240501 14:05:34.571853 5406 tablet_copy_service.cc:433] P be56773105e744919601127a94e98adf: ending tablet copy session 4360b59e3a154688a79ca5f9c0a37bc9-9b88c43cb8704583a3e65c5c35c205f6 on tablet 9b88c43cb8704583a3e65c5c35c205f6 with peer 4360b59e3a154688a79ca5f9c0a37bc9
I20240501 14:05:34.572196 8285 tablet_copy_client.cc:783] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 19a6d2d8a08142d5bffcde3a37234692: tablet copy: Starting download of 0 data blocks...
I20240501 14:05:34.572500 8285 tablet_copy_client.cc:647] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 19a6d2d8a08142d5bffcde3a37234692: tablet copy: Starting download of 1 WAL segments...
I20240501 14:05:34.578874 8549 ts_tablet_manager.cc:1875] T f9cb784f60774ea7b6e70ccfebf867a8 P 1c3fdc5272484ac09ef3e5e36a08f27c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:05:34.580395 8352 consensus_queue.cc:1035] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:34.581514 8352 raft_consensus.cc:2932] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:34.585860 4993 raft_consensus.cc:2932] T 9b88c43cb8704583a3e65c5c35c205f6 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:34.590672 6892 raft_consensus.cc:2932] T 9b88c43cb8704583a3e65c5c35c205f6 P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:34.575457 8587 ts_tablet_manager.cc:902] T 61a274b24f53458f88c2cde4f8dbe4f4 P 19a6d2d8a08142d5bffcde3a37234692: Initiating tablet copy from peer be56773105e744919601127a94e98adf (127.29.124.132:35459)
I20240501 14:05:34.600430 8590 ts_tablet_manager.cc:902] T e7d5f02ccba34254a4d20951878b1214 P 19a6d2d8a08142d5bffcde3a37234692: Initiating tablet copy from peer be56773105e744919601127a94e98adf (127.29.124.132:35459)
I20240501 14:05:34.607172 5385 consensus_queue.cc:237] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } }
I20240501 14:05:34.610316 4839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9afe3b20a43d4b5bb800af39f945948d with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240501 14:05:34.610638 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e7d5f02ccba34254a4d20951878b1214. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:34.610817 8285 tablet_copy_client.cc:523] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 19a6d2d8a08142d5bffcde3a37234692: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:05:34.614750 8285 tablet_bootstrap.cc:492] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 19a6d2d8a08142d5bffcde3a37234692: Bootstrap starting.
I20240501 14:05:34.615753 7004 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c39b53867e047faaf24cc6f87a8100c with delete_type TABLET_DATA_TOMBSTONED (TS 19a6d2d8a08142d5bffcde3a37234692 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:57226
I20240501 14:05:34.616207 4851 catalog_manager.cc:5484] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 reported cstate change: config changed from index 5 to 6, VOTER 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) evicted. New cstate: current_term: 1 leader_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:05:34.616580 8503 raft_consensus.cc:1050] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf: attempting to promote NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 to VOTER
I20240501 14:05:34.617370 8503 consensus_queue.cc:237] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } }
W20240501 14:05:34.628918 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e7d5f02ccba34254a4d20951878b1214. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:34.630039 4993 raft_consensus.cc:1261] T e7d5f02ccba34254a4d20951878b1214 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:34.630589 8352 consensus_queue.cc:1035] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:34.633083 8512 raft_consensus.cc:2932] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:34.639529 4846 catalog_manager.cc:5484] T 9b88c43cb8704583a3e65c5c35c205f6 P 938a24b0e6a040b7966225af26558ff8 reported cstate change: config changed from index 5 to 6, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:34.639802 4993 raft_consensus.cc:2932] T e7d5f02ccba34254a4d20951878b1214 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:34.643613 7025 raft_consensus.cc:1261] T f4d2b8c005b64bbb8e6621da068d15e5 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 LEARNER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:34.644358 8503 consensus_queue.cc:1035] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:34.645447 5124 raft_consensus.cc:1261] T f4d2b8c005b64bbb8e6621da068d15e5 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:34.646299 8352 consensus_queue.cc:1035] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:34.648458 4846 catalog_manager.cc:5484] T e7d5f02ccba34254a4d20951878b1214 P 04088ee3bea440a7bd5ff772152b9410 reported cstate change: config changed from index 4 to 5, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
W20240501 14:05:34.659749 4840 catalog_manager.cc:4797] TS 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): delete failed for tablet 2c39b53867e047faaf24cc6f87a8100c because the tablet was not found. No further retry: Not found: Tablet not found: 2c39b53867e047faaf24cc6f87a8100c
I20240501 14:05:34.579516 5252 raft_consensus.cc:1261] T 61a274b24f53458f88c2cde4f8dbe4f4 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:34.630791 5236 tablet_service.cc:1514] Processing DeleteTablet for tablet 9afe3b20a43d4b5bb800af39f945948d with delete_type TABLET_DATA_TOMBSTONED (TS 1c3fdc5272484ac09ef3e5e36a08f27c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39020
I20240501 14:05:34.643369 5256 raft_consensus.cc:1261] T e7d5f02ccba34254a4d20951878b1214 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:34.652376 5251 raft_consensus.cc:1261] T f4d2b8c005b64bbb8e6621da068d15e5 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:34.666014 8503 consensus_queue.cc:1035] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:34.666297 8352 consensus_queue.cc:1035] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:34.665764 8599 consensus_queue.cc:1035] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:34.673246 5252 raft_consensus.cc:2932] T 61a274b24f53458f88c2cde4f8dbe4f4 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:34.677685 8352 raft_consensus.cc:2932] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:34.678803 7025 raft_consensus.cc:2932] T f4d2b8c005b64bbb8e6621da068d15e5 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:34.673246 5251 raft_consensus.cc:2932] T e7d5f02ccba34254a4d20951878b1214 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:34.683387 5124 raft_consensus.cc:2932] T f4d2b8c005b64bbb8e6621da068d15e5 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:34.674693 5385 consensus_queue.cc:237] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }
I20240501 14:05:34.689833 4851 catalog_manager.cc:5484] T f4d2b8c005b64bbb8e6621da068d15e5 P e9c60449cf3c46f8aca4ab6c535302a4 reported cstate change: config changed from index 4 to 5, 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:34.695874 8590 tablet_copy_client.cc:308] T e7d5f02ccba34254a4d20951878b1214 P 19a6d2d8a08142d5bffcde3a37234692: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.132:35459
I20240501 14:05:34.696872 5406 tablet_copy_service.cc:140] P be56773105e744919601127a94e98adf: Received BeginTabletCopySession request for tablet e7d5f02ccba34254a4d20951878b1214 from peer 19a6d2d8a08142d5bffcde3a37234692 ({username='slave'} at 127.29.124.134:56125)
I20240501 14:05:34.697054 5406 tablet_copy_service.cc:161] P be56773105e744919601127a94e98adf: Beginning new tablet copy session on tablet e7d5f02ccba34254a4d20951878b1214 from peer 19a6d2d8a08142d5bffcde3a37234692 at {username='slave'} at 127.29.124.134:56125: session id = 19a6d2d8a08142d5bffcde3a37234692-e7d5f02ccba34254a4d20951878b1214
W20240501 14:05:34.701445 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e7d5f02ccba34254a4d20951878b1214. This is attempt 1: this message will repeat every 5th retry.
W20240501 14:05:34.702015 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e7d5f02ccba34254a4d20951878b1214. This is attempt 1: this message will repeat every 5th retry.
W20240501 14:05:34.703457 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e7d5f02ccba34254a4d20951878b1214. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:34.704442 4993 raft_consensus.cc:1261] T e7d5f02ccba34254a4d20951878b1214 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:34.704645 5406 tablet_copy_source_session.cc:205] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:05:34.704789 5276 tablet_copy_service.cc:140] P 1c3fdc5272484ac09ef3e5e36a08f27c: Received BeginTabletCopySession request for tablet 71660575d94748d8a9329f909a42a0cd from peer 938a24b0e6a040b7966225af26558ff8 ({username='slave'} at 127.29.124.133:49097)
I20240501 14:05:34.704924 5276 tablet_copy_service.cc:161] P 1c3fdc5272484ac09ef3e5e36a08f27c: Beginning new tablet copy session on tablet 71660575d94748d8a9329f909a42a0cd from peer 938a24b0e6a040b7966225af26558ff8 at {username='slave'} at 127.29.124.133:49097: session id = 938a24b0e6a040b7966225af26558ff8-71660575d94748d8a9329f909a42a0cd
I20240501 14:05:34.705130 8247 consensus_queue.cc:1035] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:34.706970 8590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7d5f02ccba34254a4d20951878b1214. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:34.707557 5276 tablet_copy_source_session.cc:205] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:05:34.707736 8247 raft_consensus.cc:2932] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
W20240501 14:05:34.708182 5191 consensus_peers.cc:485] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 51cbcf4888c440c2ad096c988ab2d968. This is attempt 1: this message will repeat every 5th retry.
W20240501 14:05:34.709025 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 71660575d94748d8a9329f909a42a0cd. This is attempt 1: this message will repeat every 5th retry.
W20240501 14:05:34.709839 5191 consensus_peers.cc:485] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 51cbcf4888c440c2ad096c988ab2d968. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:34.712754 4993 raft_consensus.cc:2932] T e7d5f02ccba34254a4d20951878b1214 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:34.720293 8227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71660575d94748d8a9329f909a42a0cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:34.720463 8590 tablet_copy_client.cc:783] T e7d5f02ccba34254a4d20951878b1214 P 19a6d2d8a08142d5bffcde3a37234692: tablet copy: Starting download of 0 data blocks...
I20240501 14:05:34.720712 8590 tablet_copy_client.cc:647] T e7d5f02ccba34254a4d20951878b1214 P 19a6d2d8a08142d5bffcde3a37234692: tablet copy: Starting download of 1 WAL segments...
I20240501 14:05:34.724896 4853 catalog_manager.cc:5484] T e7d5f02ccba34254a4d20951878b1214 P 04088ee3bea440a7bd5ff772152b9410 reported cstate change: config changed from index 5 to 6, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:34.713284 5256 raft_consensus.cc:2932] T f4d2b8c005b64bbb8e6621da068d15e5 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:34.727416 8590 tablet_copy_client.cc:523] T e7d5f02ccba34254a4d20951878b1214 P 19a6d2d8a08142d5bffcde3a37234692: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:05:34.727435 8227 tablet_copy_client.cc:783] T 71660575d94748d8a9329f909a42a0cd P 938a24b0e6a040b7966225af26558ff8: tablet copy: Starting download of 0 data blocks...
I20240501 14:05:34.727694 8227 tablet_copy_client.cc:647] T 71660575d94748d8a9329f909a42a0cd P 938a24b0e6a040b7966225af26558ff8: tablet copy: Starting download of 1 WAL segments...
I20240501 14:05:34.728989 8488 raft_consensus.cc:1050] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c: attempting to promote NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 to VOTER
I20240501 14:05:34.730110 8488 consensus_queue.cc:237] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } }
I20240501 14:05:34.731539 8590 tablet_bootstrap.cc:492] T e7d5f02ccba34254a4d20951878b1214 P 19a6d2d8a08142d5bffcde3a37234692: Bootstrap starting.
I20240501 14:05:34.743892 8227 tablet_copy_client.cc:523] T 71660575d94748d8a9329f909a42a0cd P 938a24b0e6a040b7966225af26558ff8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:05:34.745512 4995 raft_consensus.cc:1261] T 9afe3b20a43d4b5bb800af39f945948d P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:34.745798 8285 tablet_bootstrap.cc:492] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 19a6d2d8a08142d5bffcde3a37234692: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:05:34.746296 8285 tablet_bootstrap.cc:492] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 19a6d2d8a08142d5bffcde3a37234692: Bootstrap complete.
I20240501 14:05:34.746477 8383 consensus_queue.cc:1035] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:34.746625 8285 ts_tablet_manager.cc:1367] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 19a6d2d8a08142d5bffcde3a37234692: Time spent bootstrapping tablet: real 0.132s user 0.042s sys 0.008s
I20240501 14:05:34.747850 8285 raft_consensus.cc:348] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } }
I20240501 14:05:34.748093 8285 raft_consensus.cc:729] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 19a6d2d8a08142d5bffcde3a37234692, State: Initialized, Role: LEARNER
I20240501 14:05:34.748654 8285 consensus_queue.cc:260] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 19a6d2d8a08142d5bffcde3a37234692 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } }
I20240501 14:05:34.748798 4995 raft_consensus.cc:2932] T 9afe3b20a43d4b5bb800af39f945948d P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:34.749297 8285 ts_tablet_manager.cc:1398] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 19a6d2d8a08142d5bffcde3a37234692: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240501 14:05:34.753736 5276 tablet_copy_service.cc:342] P 1c3fdc5272484ac09ef3e5e36a08f27c: Request end of tablet copy session 19a6d2d8a08142d5bffcde3a37234692-23ca2bf1e59f44fcac8a48aff8d056b1 received from {username='slave'} at 127.29.124.134:43435
I20240501 14:05:34.753927 5276 tablet_copy_service.cc:433] P 1c3fdc5272484ac09ef3e5e36a08f27c: ending tablet copy session 19a6d2d8a08142d5bffcde3a37234692-23ca2bf1e59f44fcac8a48aff8d056b1 on tablet 23ca2bf1e59f44fcac8a48aff8d056b1 with peer 19a6d2d8a08142d5bffcde3a37234692
I20240501 14:05:34.755383 5254 consensus_queue.cc:237] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } }
I20240501 14:05:34.757417 8594 ts_tablet_manager.cc:902] T 61a274b24f53458f88c2cde4f8dbe4f4 P 6dac90f01e744f9c8701a404440a4c77: Initiating tablet copy from peer be56773105e744919601127a94e98adf (127.29.124.132:35459)
I20240501 14:05:34.758090 5124 raft_consensus.cc:1261] T ec1e81011c6a46bc8a39aecf3a5e71e4 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:05:34.758654 8281 consensus_queue.cc:1035] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:05:34.759927 5252 raft_consensus.cc:1261] T e7d5f02ccba34254a4d20951878b1214 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:34.760571 8503 consensus_queue.cc:1035] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:34.760411 5255 raft_consensus.cc:2932] T 9b88c43cb8704583a3e65c5c35c205f6 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:34.763295 5254 raft_consensus.cc:2932] T e7d5f02ccba34254a4d20951878b1214 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:34.771029 4995 raft_consensus.cc:1261] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:05:34.771677 8227 tablet_bootstrap.cc:492] T 71660575d94748d8a9329f909a42a0cd P 938a24b0e6a040b7966225af26558ff8: Bootstrap starting.
I20240501 14:05:34.771829 8277 consensus_queue.cc:1035] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:05:34.776548 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ec1e81011c6a46bc8a39aecf3a5e71e4. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:34.790845 5254 consensus_queue.cc:237] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } }
I20240501 14:05:34.794230 5385 consensus_queue.cc:237] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } }
I20240501 14:05:34.796649 7024 raft_consensus.cc:1261] T f4d2b8c005b64bbb8e6621da068d15e5 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:34.798391 8352 consensus_queue.cc:1035] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:34.801059 8352 raft_consensus.cc:2932] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:34.801128 5385 consensus_queue.cc:237] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:05:34.805099 4839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f4d2b8c005b64bbb8e6621da068d15e5 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:05:34.805526 7024 raft_consensus.cc:2932] T f4d2b8c005b64bbb8e6621da068d15e5 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:34.805802 6892 raft_consensus.cc:1261] T 9b88c43cb8704583a3e65c5c35c205f6 P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:05:34.807062 5255 raft_consensus.cc:1261] T 9b88c43cb8704583a3e65c5c35c205f6 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:05:34.808800 8612 consensus_queue.cc:1035] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:05:34.809753 5126 raft_consensus.cc:1261] T 404e2851055541f0be1bb0971a7cc72a P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:05:34.810371 8281 consensus_queue.cc:1035] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:05:34.811102 4853 catalog_manager.cc:5484] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf reported cstate change: config changed from index 5 to 6, VOTER e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130) evicted. New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 14:05:34.812814 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 404e2851055541f0be1bb0971a7cc72a. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:34.813138 8352 consensus_queue.cc:1035] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:05:34.816066 5106 tablet_service.cc:1514] Processing DeleteTablet for tablet f4d2b8c005b64bbb8e6621da068d15e5 with delete_type TABLET_DATA_TOMBSTONED (TS e9c60449cf3c46f8aca4ab6c535302a4 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41850
I20240501 14:05:34.828212 5385 raft_consensus.cc:1261] T 404e2851055541f0be1bb0971a7cc72a P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:05:34.829342 8281 consensus_queue.cc:1035] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:05:34.832378 8352 raft_consensus.cc:2932] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:34.837416 5255 raft_consensus.cc:2932] T 9b88c43cb8704583a3e65c5c35c205f6 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:34.841760 8629 tablet_replica.cc:330] T f4d2b8c005b64bbb8e6621da068d15e5 P e9c60449cf3c46f8aca4ab6c535302a4: stopping tablet replica
I20240501 14:05:34.842135 8629 raft_consensus.cc:2229] T f4d2b8c005b64bbb8e6621da068d15e5 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:05:34.842469 8629 raft_consensus.cc:2258] T f4d2b8c005b64bbb8e6621da068d15e5 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:05:34.844491 4839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9b88c43cb8704583a3e65c5c35c205f6 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:05:34.849023 4846 catalog_manager.cc:5484] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf reported cstate change: config changed from index 6 to 7, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) evicted. New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:05:34.852527 6892 raft_consensus.cc:2932] T 9b88c43cb8704583a3e65c5c35c205f6 P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:34.859139 7025 raft_consensus.cc:1203] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:05:34.859241 8241 ts_tablet_manager.cc:902] T 51cbcf4888c440c2ad096c988ab2d968 P 4360b59e3a154688a79ca5f9c0a37bc9: Initiating tablet copy from peer 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209)
I20240501 14:05:34.863601 8549 ts_tablet_manager.cc:1888] T f9cb784f60774ea7b6e70ccfebf867a8 P 1c3fdc5272484ac09ef3e5e36a08f27c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:05:34.866705 8629 ts_tablet_manager.cc:1875] T f4d2b8c005b64bbb8e6621da068d15e5 P e9c60449cf3c46f8aca4ab6c535302a4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:05:34.871991 8549 log.cc:1196] T f9cb784f60774ea7b6e70ccfebf867a8 P 1c3fdc5272484ac09ef3e5e36a08f27c: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/wal/wals/f9cb784f60774ea7b6e70ccfebf867a8
I20240501 14:05:34.878821 8500 raft_consensus.cc:2932] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:34.880996 8549 tablet_replica.cc:330] T 2c39b53867e047faaf24cc6f87a8100c P 1c3fdc5272484ac09ef3e5e36a08f27c: stopping tablet replica
I20240501 14:05:34.881565 8549 raft_consensus.cc:2229] T 2c39b53867e047faaf24cc6f87a8100c P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:05:34.883224 4838 catalog_manager.cc:4830] TS 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209): tablet f9cb784f60774ea7b6e70ccfebf867a8 (table rebalance_test_table_5 [id=3794b1ccb5004f7cb92a229ef6ce3595]) successfully deleted
I20240501 14:05:34.883592 8549 raft_consensus.cc:2258] T 2c39b53867e047faaf24cc6f87a8100c P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:05:34.883992 5125 raft_consensus.cc:2932] T ec1e81011c6a46bc8a39aecf3a5e71e4 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:34.886037 8549 ts_tablet_manager.cc:1875] T 2c39b53867e047faaf24cc6f87a8100c P 1c3fdc5272484ac09ef3e5e36a08f27c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:05:34.889992 5385 consensus_queue.cc:237] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:05:34.892343 5255 raft_consensus.cc:1261] T 9b88c43cb8704583a3e65c5c35c205f6 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:05:34.893007 8503 consensus_queue.cc:1035] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:05:34.893363 6891 raft_consensus.cc:1261] T 9b88c43cb8704583a3e65c5c35c205f6 P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:05:34.895289 8503 consensus_queue.cc:1035] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:05:34.896023 8503 raft_consensus.cc:2932] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:34.896521 4851 catalog_manager.cc:5484] T ec1e81011c6a46bc8a39aecf3a5e71e4 P e9c60449cf3c46f8aca4ab6c535302a4 reported cstate change: config changed from index -1 to 4, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) added. New cstate: current_term: 1 leader_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:34.896639 5255 raft_consensus.cc:2932] T 9b88c43cb8704583a3e65c5c35c205f6 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:34.899899 6890 raft_consensus.cc:2932] T 9b88c43cb8704583a3e65c5c35c205f6 P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:34.904939 8642 ts_tablet_manager.cc:902] T 51cbcf4888c440c2ad096c988ab2d968 P 938a24b0e6a040b7966225af26558ff8: Initiating tablet copy from peer 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209)
I20240501 14:05:34.911293 5256 raft_consensus.cc:1261] T f4d2b8c005b64bbb8e6621da068d15e5 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:34.911974 4839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9b88c43cb8704583a3e65c5c35c205f6 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:05:34.913463 8352 consensus_queue.cc:1035] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:34.916502 4851 catalog_manager.cc:5484] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf reported cstate change: config changed from index 7 to 8, VOTER 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129) evicted. New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:05:34.916944 8629 ts_tablet_manager.cc:1888] T f4d2b8c005b64bbb8e6621da068d15e5 P e9c60449cf3c46f8aca4ab6c535302a4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:05:34.917081 8629 log.cc:1196] T f4d2b8c005b64bbb8e6621da068d15e5 P e9c60449cf3c46f8aca4ab6c535302a4: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/wal/wals/f4d2b8c005b64bbb8e6621da068d15e5
I20240501 14:05:34.917879 4839 catalog_manager.cc:4830] TS e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377): tablet f4d2b8c005b64bbb8e6621da068d15e5 (table rebalance_test_table_1 [id=403fdb578f8e4066a56f685925e83936]) successfully deleted
I20240501 14:05:34.927052 5256 raft_consensus.cc:2932] T f4d2b8c005b64bbb8e6621da068d15e5 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:34.927842 4995 raft_consensus.cc:1261] T 9b88c43cb8704583a3e65c5c35c205f6 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:05:34.928395 7025 raft_consensus.cc:1261] T 1a7bbfc15a76450bbc9197037e856e08 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 LEARNER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:34.929216 8500 consensus_queue.cc:1035] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:34.929425 5385 raft_consensus.cc:1261] T 1a7bbfc15a76450bbc9197037e856e08 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:34.929980 8500 consensus_queue.cc:1035] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:34.947237 5124 raft_consensus.cc:1261] T 1a7bbfc15a76450bbc9197037e856e08 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:34.948484 8488 consensus_queue.cc:1035] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:34.958379 8500 raft_consensus.cc:2932] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } }
I20240501 14:05:34.961742 8642 tablet_copy_client.cc:308] T 51cbcf4888c440c2ad096c988ab2d968 P 938a24b0e6a040b7966225af26558ff8: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.131:33209
I20240501 14:05:34.964733 5276 tablet_copy_service.cc:140] P 1c3fdc5272484ac09ef3e5e36a08f27c: Received BeginTabletCopySession request for tablet 51cbcf4888c440c2ad096c988ab2d968 from peer 938a24b0e6a040b7966225af26558ff8 ({username='slave'} at 127.29.124.133:49097)
I20240501 14:05:34.964895 5276 tablet_copy_service.cc:161] P 1c3fdc5272484ac09ef3e5e36a08f27c: Beginning new tablet copy session on tablet 51cbcf4888c440c2ad096c988ab2d968 from peer 938a24b0e6a040b7966225af26558ff8 at {username='slave'} at 127.29.124.133:49097: session id = 938a24b0e6a040b7966225af26558ff8-51cbcf4888c440c2ad096c988ab2d968
I20240501 14:05:34.966131 8549 ts_tablet_manager.cc:1888] T 2c39b53867e047faaf24cc6f87a8100c P 1c3fdc5272484ac09ef3e5e36a08f27c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:05:34.966274 8549 log.cc:1196] T 2c39b53867e047faaf24cc6f87a8100c P 1c3fdc5272484ac09ef3e5e36a08f27c: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/wal/wals/2c39b53867e047faaf24cc6f87a8100c
I20240501 14:05:34.967705 5276 tablet_copy_source_session.cc:205] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:05:34.968420 4838 catalog_manager.cc:4830] TS 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209): tablet 2c39b53867e047faaf24cc6f87a8100c (table rebalance_test_table_5 [id=3794b1ccb5004f7cb92a229ef6ce3595]) successfully deleted
I20240501 14:05:34.969115 8277 raft_consensus.cc:2932] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:34.972362 8549 tablet_replica.cc:330] T 9afe3b20a43d4b5bb800af39f945948d P 1c3fdc5272484ac09ef3e5e36a08f27c: stopping tablet replica
I20240501 14:05:34.972617 8549 raft_consensus.cc:2229] T 9afe3b20a43d4b5bb800af39f945948d P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:05:34.972987 8549 raft_consensus.cc:2258] T 9afe3b20a43d4b5bb800af39f945948d P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:05:34.973968 5384 raft_consensus.cc:2932] T 1a7bbfc15a76450bbc9197037e856e08 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:34.980008 4853 catalog_manager.cc:5484] T 1a7bbfc15a76450bbc9197037e856e08 P be56773105e744919601127a94e98adf reported cstate change: config changed from index 4 to 5, 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:34.980943 8549 ts_tablet_manager.cc:1875] T 9afe3b20a43d4b5bb800af39f945948d P 1c3fdc5272484ac09ef3e5e36a08f27c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:05:34.984632 8557 tablet_copy_client.cc:308] T ba6cba753e934e719c730593c0706346 P 4360b59e3a154688a79ca5f9c0a37bc9: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.129:44033
I20240501 14:05:34.985239 8587 tablet_copy_client.cc:308] T 61a274b24f53458f88c2cde4f8dbe4f4 P 19a6d2d8a08142d5bffcde3a37234692: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.132:35459
I20240501 14:05:34.985445 5015 tablet_copy_service.cc:140] P 04088ee3bea440a7bd5ff772152b9410: Received BeginTabletCopySession request for tablet ba6cba753e934e719c730593c0706346 from peer 4360b59e3a154688a79ca5f9c0a37bc9 ({username='slave'} at 127.29.124.135:54069)
I20240501 14:05:34.985599 5015 tablet_copy_service.cc:161] P 04088ee3bea440a7bd5ff772152b9410: Beginning new tablet copy session on tablet ba6cba753e934e719c730593c0706346 from peer 4360b59e3a154688a79ca5f9c0a37bc9 at {username='slave'} at 127.29.124.135:54069: session id = 4360b59e3a154688a79ca5f9c0a37bc9-ba6cba753e934e719c730593c0706346
I20240501 14:05:34.985898 5124 raft_consensus.cc:2932] T 1a7bbfc15a76450bbc9197037e856e08 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:34.987403 5125 raft_consensus.cc:2932] T 404e2851055541f0be1bb0971a7cc72a P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } }
I20240501 14:05:34.988445 5015 tablet_copy_source_session.cc:205] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:05:34.989694 5406 tablet_copy_service.cc:140] P be56773105e744919601127a94e98adf: Received BeginTabletCopySession request for tablet 61a274b24f53458f88c2cde4f8dbe4f4 from peer 19a6d2d8a08142d5bffcde3a37234692 ({username='slave'} at 127.29.124.134:56125)
I20240501 14:05:34.989817 8557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba6cba753e934e719c730593c0706346. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 14:05:34.989830 4928 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.124.131:33209, user_credentials={real_user=slave}} blocked reactor thread for 181481us
I20240501 14:05:34.989899 5406 tablet_copy_service.cc:161] P be56773105e744919601127a94e98adf: Beginning new tablet copy session on tablet 61a274b24f53458f88c2cde4f8dbe4f4 from peer 19a6d2d8a08142d5bffcde3a37234692 at {username='slave'} at 127.29.124.134:56125: session id = 19a6d2d8a08142d5bffcde3a37234692-61a274b24f53458f88c2cde4f8dbe4f4
I20240501 14:05:34.990749 7136 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b88c43cb8704583a3e65c5c35c205f6 with delete_type TABLET_DATA_TOMBSTONED (TS 4360b59e3a154688a79ca5f9c0a37bc9 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:51170
I20240501 14:05:34.992724 5406 tablet_copy_source_session.cc:205] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:05:34.994195 7025 raft_consensus.cc:2932] T 1a7bbfc15a76450bbc9197037e856e08 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:35.000195 8587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61a274b24f53458f88c2cde4f8dbe4f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:35.001073 8642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51cbcf4888c440c2ad096c988ab2d968. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 14:05:34.990206 4929 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.124.130:34377, user_credentials={real_user=slave}} blocked reactor thread for 203376us
I20240501 14:05:35.005604 4973 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b88c43cb8704583a3e65c5c35c205f6 with delete_type TABLET_DATA_TOMBSTONED (TS 04088ee3bea440a7bd5ff772152b9410 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:41194
I20240501 14:05:35.011472 8662 tablet_replica.cc:330] T 9b88c43cb8704583a3e65c5c35c205f6 P 4360b59e3a154688a79ca5f9c0a37bc9: stopping tablet replica
I20240501 14:05:35.011775 8662 raft_consensus.cc:2229] T 9b88c43cb8704583a3e65c5c35c205f6 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 LEARNER]: Raft consensus shutting down.
I20240501 14:05:35.015718 8662 raft_consensus.cc:2258] T 9b88c43cb8704583a3e65c5c35c205f6 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 LEARNER]: Raft consensus is shut down!
I20240501 14:05:35.017693 5382 raft_consensus.cc:2932] T 404e2851055541f0be1bb0971a7cc72a P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } }
I20240501 14:05:35.018543 8241 tablet_copy_client.cc:308] T 51cbcf4888c440c2ad096c988ab2d968 P 4360b59e3a154688a79ca5f9c0a37bc9: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.131:33209
I20240501 14:05:35.020153 8662 ts_tablet_manager.cc:1875] T 9b88c43cb8704583a3e65c5c35c205f6 P 4360b59e3a154688a79ca5f9c0a37bc9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:05:35.020730 5276 tablet_copy_service.cc:140] P 1c3fdc5272484ac09ef3e5e36a08f27c: Received BeginTabletCopySession request for tablet 51cbcf4888c440c2ad096c988ab2d968 from peer 4360b59e3a154688a79ca5f9c0a37bc9 ({username='slave'} at 127.29.124.135:51325)
I20240501 14:05:35.021021 5276 tablet_copy_service.cc:161] P 1c3fdc5272484ac09ef3e5e36a08f27c: Beginning new tablet copy session on tablet 51cbcf4888c440c2ad096c988ab2d968 from peer 4360b59e3a154688a79ca5f9c0a37bc9 at {username='slave'} at 127.29.124.135:51325: session id = 4360b59e3a154688a79ca5f9c0a37bc9-51cbcf4888c440c2ad096c988ab2d968
I20240501 14:05:35.024713 5276 tablet_copy_source_session.cc:205] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:05:35.026046 8241 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 51cbcf4888c440c2ad096c988ab2d968. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:35.027805 8549 ts_tablet_manager.cc:1888] T 9afe3b20a43d4b5bb800af39f945948d P 1c3fdc5272484ac09ef3e5e36a08f27c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.4
I20240501 14:05:35.029528 8590 tablet_bootstrap.cc:492] T e7d5f02ccba34254a4d20951878b1214 P 19a6d2d8a08142d5bffcde3a37234692: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 14:05:35.029608 8549 log.cc:1196] T 9afe3b20a43d4b5bb800af39f945948d P 1c3fdc5272484ac09ef3e5e36a08f27c: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/wal/wals/9afe3b20a43d4b5bb800af39f945948d
I20240501 14:05:35.030258 8590 tablet_bootstrap.cc:492] T e7d5f02ccba34254a4d20951878b1214 P 19a6d2d8a08142d5bffcde3a37234692: Bootstrap complete.
I20240501 14:05:35.032305 8590 ts_tablet_manager.cc:1367] T e7d5f02ccba34254a4d20951878b1214 P 19a6d2d8a08142d5bffcde3a37234692: Time spent bootstrapping tablet: real 0.301s user 0.033s sys 0.046s
I20240501 14:05:35.032765 8241 tablet_copy_client.cc:783] T 51cbcf4888c440c2ad096c988ab2d968 P 4360b59e3a154688a79ca5f9c0a37bc9: tablet copy: Starting download of 0 data blocks...
I20240501 14:05:35.032995 8241 tablet_copy_client.cc:647] T 51cbcf4888c440c2ad096c988ab2d968 P 4360b59e3a154688a79ca5f9c0a37bc9: tablet copy: Starting download of 1 WAL segments...
I20240501 14:05:35.033768 8590 raft_consensus.cc:348] T e7d5f02ccba34254a4d20951878b1214 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } }
I20240501 14:05:35.034560 4838 catalog_manager.cc:4830] TS 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209): tablet 9afe3b20a43d4b5bb800af39f945948d (table rebalance_test_table_3 [id=41272b857ab045629ca60ab2467097b0]) successfully deleted
I20240501 14:05:35.038059 8590 raft_consensus.cc:729] T e7d5f02ccba34254a4d20951878b1214 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 19a6d2d8a08142d5bffcde3a37234692, State: Initialized, Role: LEARNER
I20240501 14:05:35.038898 8594 tablet_copy_client.cc:308] T 61a274b24f53458f88c2cde4f8dbe4f4 P 6dac90f01e744f9c8701a404440a4c77: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.132:35459
I20240501 14:05:35.039379 4992 raft_consensus.cc:2932] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:35.044924 8557 tablet_copy_client.cc:783] T ba6cba753e934e719c730593c0706346 P 4360b59e3a154688a79ca5f9c0a37bc9: tablet copy: Starting download of 0 data blocks...
I20240501 14:05:35.045185 8557 tablet_copy_client.cc:647] T ba6cba753e934e719c730593c0706346 P 4360b59e3a154688a79ca5f9c0a37bc9: tablet copy: Starting download of 1 WAL segments...
I20240501 14:05:35.050971 8590 consensus_queue.cc:260] T e7d5f02ccba34254a4d20951878b1214 P 19a6d2d8a08142d5bffcde3a37234692 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }
I20240501 14:05:35.054538 8587 tablet_copy_client.cc:783] T 61a274b24f53458f88c2cde4f8dbe4f4 P 19a6d2d8a08142d5bffcde3a37234692: tablet copy: Starting download of 0 data blocks...
I20240501 14:05:35.054970 8587 tablet_copy_client.cc:647] T 61a274b24f53458f88c2cde4f8dbe4f4 P 19a6d2d8a08142d5bffcde3a37234692: tablet copy: Starting download of 1 WAL segments...
I20240501 14:05:35.055069 8590 ts_tablet_manager.cc:1398] T e7d5f02ccba34254a4d20951878b1214 P 19a6d2d8a08142d5bffcde3a37234692: Time spent starting tablet: real 0.022s user 0.000s sys 0.002s
I20240501 14:05:35.060281 8241 tablet_copy_client.cc:523] T 51cbcf4888c440c2ad096c988ab2d968 P 4360b59e3a154688a79ca5f9c0a37bc9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:05:35.063829 8241 tablet_bootstrap.cc:492] T 51cbcf4888c440c2ad096c988ab2d968 P 4360b59e3a154688a79ca5f9c0a37bc9: Bootstrap starting.
I20240501 14:05:35.066958 8557 tablet_copy_client.cc:523] T ba6cba753e934e719c730593c0706346 P 4360b59e3a154688a79ca5f9c0a37bc9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:05:35.070674 8557 tablet_bootstrap.cc:492] T ba6cba753e934e719c730593c0706346 P 4360b59e3a154688a79ca5f9c0a37bc9: Bootstrap starting.
I20240501 14:05:35.072183 8409 tablet_bootstrap.cc:492] T 3d6859d53fd747a5a5e94722626fa522 P 938a24b0e6a040b7966225af26558ff8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:05:35.073369 8409 tablet_bootstrap.cc:492] T 3d6859d53fd747a5a5e94722626fa522 P 938a24b0e6a040b7966225af26558ff8: Bootstrap complete.
I20240501 14:05:35.073725 8409 ts_tablet_manager.cc:1367] T 3d6859d53fd747a5a5e94722626fa522 P 938a24b0e6a040b7966225af26558ff8: Time spent bootstrapping tablet: real 0.680s user 0.061s sys 0.015s
I20240501 14:05:35.074867 8663 tablet_replica.cc:330] T 9b88c43cb8704583a3e65c5c35c205f6 P 04088ee3bea440a7bd5ff772152b9410: stopping tablet replica
I20240501 14:05:35.075217 8663 raft_consensus.cc:2229] T 9b88c43cb8704583a3e65c5c35c205f6 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:05:35.075098 8409 raft_consensus.cc:348] T 3d6859d53fd747a5a5e94722626fa522 P 938a24b0e6a040b7966225af26558ff8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }
I20240501 14:05:35.075346 8409 raft_consensus.cc:729] T 3d6859d53fd747a5a5e94722626fa522 P 938a24b0e6a040b7966225af26558ff8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 938a24b0e6a040b7966225af26558ff8, State: Initialized, Role: LEARNER
I20240501 14:05:35.075603 8663 raft_consensus.cc:2258] T 9b88c43cb8704583a3e65c5c35c205f6 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:05:35.075644 8409 consensus_queue.cc:260] T 3d6859d53fd747a5a5e94722626fa522 P 938a24b0e6a040b7966225af26558ff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }
I20240501 14:05:35.077214 8409 ts_tablet_manager.cc:1398] T 3d6859d53fd747a5a5e94722626fa522 P 938a24b0e6a040b7966225af26558ff8: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240501 14:05:35.078485 5146 tablet_copy_service.cc:342] P e9c60449cf3c46f8aca4ab6c535302a4: Request end of tablet copy session 938a24b0e6a040b7966225af26558ff8-3d6859d53fd747a5a5e94722626fa522 received from {username='slave'} at 127.29.124.133:48619
I20240501 14:05:35.078632 5146 tablet_copy_service.cc:433] P e9c60449cf3c46f8aca4ab6c535302a4: ending tablet copy session 938a24b0e6a040b7966225af26558ff8-3d6859d53fd747a5a5e94722626fa522 on tablet 3d6859d53fd747a5a5e94722626fa522 with peer 938a24b0e6a040b7966225af26558ff8
I20240501 14:05:35.080917 8587 tablet_copy_client.cc:523] T 61a274b24f53458f88c2cde4f8dbe4f4 P 19a6d2d8a08142d5bffcde3a37234692: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:05:35.081634 5406 tablet_copy_service.cc:342] P be56773105e744919601127a94e98adf: Request end of tablet copy session 19a6d2d8a08142d5bffcde3a37234692-e7d5f02ccba34254a4d20951878b1214 received from {username='slave'} at 127.29.124.134:56125
I20240501 14:05:35.081766 5406 tablet_copy_service.cc:433] P be56773105e744919601127a94e98adf: ending tablet copy session 19a6d2d8a08142d5bffcde3a37234692-e7d5f02ccba34254a4d20951878b1214 on tablet e7d5f02ccba34254a4d20951878b1214 with peer 19a6d2d8a08142d5bffcde3a37234692
I20240501 14:05:35.084106 4853 catalog_manager.cc:5484] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c reported cstate change: config changed from index -1 to 4, NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) added. New cstate: current_term: 1 leader_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:05:35.084610 8587 tablet_bootstrap.cc:492] T 61a274b24f53458f88c2cde4f8dbe4f4 P 19a6d2d8a08142d5bffcde3a37234692: Bootstrap starting.
I20240501 14:05:35.085491 5384 consensus_queue.cc:237] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }
I20240501 14:05:35.088380 5256 raft_consensus.cc:1261] T 61a274b24f53458f88c2cde4f8dbe4f4 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:35.089769 8247 consensus_queue.cc:1035] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:35.091295 8662 ts_tablet_manager.cc:1888] T 9b88c43cb8704583a3e65c5c35c205f6 P 4360b59e3a154688a79ca5f9c0a37bc9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:05:35.091431 8662 log.cc:1196] T 9b88c43cb8704583a3e65c5c35c205f6 P 4360b59e3a154688a79ca5f9c0a37bc9: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/wal/wals/9b88c43cb8704583a3e65c5c35c205f6
W20240501 14:05:35.091697 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 61a274b24f53458f88c2cde4f8dbe4f4. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:35.094522 4840 catalog_manager.cc:4830] TS 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): tablet 9b88c43cb8704583a3e65c5c35c205f6 (table rebalance_test_table_0 [id=7af67fe405db4bffbc53cb288b8bb715]) successfully deleted
I20240501 14:05:35.097885 8663 ts_tablet_manager.cc:1875] T 9b88c43cb8704583a3e65c5c35c205f6 P 04088ee3bea440a7bd5ff772152b9410: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240501 14:05:35.098757 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 61a274b24f53458f88c2cde4f8dbe4f4. This is attempt 1: this message will repeat every 5th retry.
W20240501 14:05:35.099474 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:35.102042 5406 tablet_copy_service.cc:140] P be56773105e744919601127a94e98adf: Received BeginTabletCopySession request for tablet 61a274b24f53458f88c2cde4f8dbe4f4 from peer 6dac90f01e744f9c8701a404440a4c77 ({username='slave'} at 127.29.124.136:50719)
I20240501 14:05:35.102181 5406 tablet_copy_service.cc:161] P be56773105e744919601127a94e98adf: Beginning new tablet copy session on tablet 61a274b24f53458f88c2cde4f8dbe4f4 from peer 6dac90f01e744f9c8701a404440a4c77 at {username='slave'} at 127.29.124.136:50719: session id = 6dac90f01e744f9c8701a404440a4c77-61a274b24f53458f88c2cde4f8dbe4f4
I20240501 14:05:35.106554 8247 raft_consensus.cc:2932] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:35.116897 5256 raft_consensus.cc:2932] T 61a274b24f53458f88c2cde4f8dbe4f4 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:35.127911 8642 tablet_copy_client.cc:783] T 51cbcf4888c440c2ad096c988ab2d968 P 938a24b0e6a040b7966225af26558ff8: tablet copy: Starting download of 0 data blocks...
I20240501 14:05:35.128235 8642 tablet_copy_client.cc:647] T 51cbcf4888c440c2ad096c988ab2d968 P 938a24b0e6a040b7966225af26558ff8: tablet copy: Starting download of 1 WAL segments...
I20240501 14:05:35.129949 8642 tablet_copy_client.cc:523] T 51cbcf4888c440c2ad096c988ab2d968 P 938a24b0e6a040b7966225af26558ff8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:05:35.130303 5125 raft_consensus.cc:1261] T 61a274b24f53458f88c2cde4f8dbe4f4 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:35.131862 8612 consensus_queue.cc:1035] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:35.133545 8642 tablet_bootstrap.cc:492] T 51cbcf4888c440c2ad096c988ab2d968 P 938a24b0e6a040b7966225af26558ff8: Bootstrap starting.
I20240501 14:05:35.134388 5406 tablet_copy_source_session.cc:205] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:05:35.135874 8594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61a274b24f53458f88c2cde4f8dbe4f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:35.138558 8663 ts_tablet_manager.cc:1888] T 9b88c43cb8704583a3e65c5c35c205f6 P 04088ee3bea440a7bd5ff772152b9410: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 14:05:35.138669 8663 log.cc:1196] T 9b88c43cb8704583a3e65c5c35c205f6 P 04088ee3bea440a7bd5ff772152b9410: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/wal/wals/9b88c43cb8704583a3e65c5c35c205f6
I20240501 14:05:35.137899 5125 raft_consensus.cc:2932] T 61a274b24f53458f88c2cde4f8dbe4f4 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:35.142735 8594 tablet_copy_client.cc:783] T 61a274b24f53458f88c2cde4f8dbe4f4 P 6dac90f01e744f9c8701a404440a4c77: tablet copy: Starting download of 0 data blocks...
I20240501 14:05:35.143030 8594 tablet_copy_client.cc:647] T 61a274b24f53458f88c2cde4f8dbe4f4 P 6dac90f01e744f9c8701a404440a4c77: tablet copy: Starting download of 1 WAL segments...
I20240501 14:05:35.144296 8680 ts_tablet_manager.cc:902] T 71660575d94748d8a9329f909a42a0cd P 6dac90f01e744f9c8701a404440a4c77: Initiating tablet copy from peer 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209)
I20240501 14:05:35.144811 8594 tablet_copy_client.cc:523] T 61a274b24f53458f88c2cde4f8dbe4f4 P 6dac90f01e744f9c8701a404440a4c77: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:05:35.146389 4839 catalog_manager.cc:4830] TS 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129:44033): tablet 9b88c43cb8704583a3e65c5c35c205f6 (table rebalance_test_table_0 [id=7af67fe405db4bffbc53cb288b8bb715]) successfully deleted
I20240501 14:05:35.148541 8594 tablet_bootstrap.cc:492] T 61a274b24f53458f88c2cde4f8dbe4f4 P 6dac90f01e744f9c8701a404440a4c77: Bootstrap starting.
I20240501 14:05:35.151173 4851 catalog_manager.cc:5484] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf reported cstate change: config changed from index 5 to 6, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:05:35.162694 8680 tablet_copy_client.cc:308] T 71660575d94748d8a9329f909a42a0cd P 6dac90f01e744f9c8701a404440a4c77: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.131:33209
I20240501 14:05:35.163631 5276 tablet_copy_service.cc:140] P 1c3fdc5272484ac09ef3e5e36a08f27c: Received BeginTabletCopySession request for tablet 71660575d94748d8a9329f909a42a0cd from peer 6dac90f01e744f9c8701a404440a4c77 ({username='slave'} at 127.29.124.136:49141)
I20240501 14:05:35.163825 5276 tablet_copy_service.cc:161] P 1c3fdc5272484ac09ef3e5e36a08f27c: Beginning new tablet copy session on tablet 71660575d94748d8a9329f909a42a0cd from peer 6dac90f01e744f9c8701a404440a4c77 at {username='slave'} at 127.29.124.136:49141: session id = 6dac90f01e744f9c8701a404440a4c77-71660575d94748d8a9329f909a42a0cd
I20240501 14:05:35.164361 5256 consensus_queue.cc:237] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } }
I20240501 14:05:35.167430 5384 raft_consensus.cc:1261] T 1a7bbfc15a76450bbc9197037e856e08 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:35.167579 7025 raft_consensus.cc:1261] T 1a7bbfc15a76450bbc9197037e856e08 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:35.168035 5276 tablet_copy_source_session.cc:205] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:05:35.168005 8488 consensus_queue.cc:1035] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:35.168327 8500 consensus_queue.cc:1035] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:35.171113 8500 raft_consensus.cc:2932] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:35.171972 5384 raft_consensus.cc:2932] T 1a7bbfc15a76450bbc9197037e856e08 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:35.175140 4838 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1a7bbfc15a76450bbc9197037e856e08 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:05:35.178120 7025 raft_consensus.cc:2932] T 1a7bbfc15a76450bbc9197037e856e08 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:35.178637 4851 catalog_manager.cc:5484] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c reported cstate change: config changed from index 5 to 6, VOTER e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130) evicted. New cstate: current_term: 1 leader_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:05:35.182535 5106 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a7bbfc15a76450bbc9197037e856e08 with delete_type TABLET_DATA_TOMBSTONED (TS e9c60449cf3c46f8aca4ab6c535302a4 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41850
I20240501 14:05:35.182897 8629 tablet_replica.cc:330] T 1a7bbfc15a76450bbc9197037e856e08 P e9c60449cf3c46f8aca4ab6c535302a4: stopping tablet replica
I20240501 14:05:35.183115 8629 raft_consensus.cc:2229] T 1a7bbfc15a76450bbc9197037e856e08 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:05:35.183341 8629 raft_consensus.cc:2258] T 1a7bbfc15a76450bbc9197037e856e08 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:05:35.184691 8629 ts_tablet_manager.cc:1875] T 1a7bbfc15a76450bbc9197037e856e08 P e9c60449cf3c46f8aca4ab6c535302a4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:05:35.185575 8680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71660575d94748d8a9329f909a42a0cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:35.192821 8629 ts_tablet_manager.cc:1888] T 1a7bbfc15a76450bbc9197037e856e08 P e9c60449cf3c46f8aca4ab6c535302a4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:05:35.193006 8629 log.cc:1196] T 1a7bbfc15a76450bbc9197037e856e08 P e9c60449cf3c46f8aca4ab6c535302a4: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/wal/wals/1a7bbfc15a76450bbc9197037e856e08
I20240501 14:05:35.193877 4839 catalog_manager.cc:4830] TS e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377): tablet 1a7bbfc15a76450bbc9197037e856e08 (table rebalance_test_table_1 [id=403fdb578f8e4066a56f685925e83936]) successfully deleted
I20240501 14:05:35.202458 8685 ts_tablet_manager.cc:902] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 4360b59e3a154688a79ca5f9c0a37bc9: Initiating tablet copy from peer 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209)
I20240501 14:05:35.207293 8227 tablet_bootstrap.cc:492] T 71660575d94748d8a9329f909a42a0cd P 938a24b0e6a040b7966225af26558ff8: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:05:35.207780 8227 tablet_bootstrap.cc:492] T 71660575d94748d8a9329f909a42a0cd P 938a24b0e6a040b7966225af26558ff8: Bootstrap complete.
I20240501 14:05:35.208353 8227 ts_tablet_manager.cc:1367] T 71660575d94748d8a9329f909a42a0cd P 938a24b0e6a040b7966225af26558ff8: Time spent bootstrapping tablet: real 0.437s user 0.040s sys 0.008s
I20240501 14:05:35.210647 8680 tablet_copy_client.cc:783] T 71660575d94748d8a9329f909a42a0cd P 6dac90f01e744f9c8701a404440a4c77: tablet copy: Starting download of 0 data blocks...
I20240501 14:05:35.211136 8680 tablet_copy_client.cc:647] T 71660575d94748d8a9329f909a42a0cd P 6dac90f01e744f9c8701a404440a4c77: tablet copy: Starting download of 1 WAL segments...
I20240501 14:05:35.212411 8227 raft_consensus.cc:348] T 71660575d94748d8a9329f909a42a0cd P 938a24b0e6a040b7966225af26558ff8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } }
I20240501 14:05:35.212687 8227 raft_consensus.cc:729] T 71660575d94748d8a9329f909a42a0cd P 938a24b0e6a040b7966225af26558ff8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 938a24b0e6a040b7966225af26558ff8, State: Initialized, Role: LEARNER
I20240501 14:05:35.212913 8227 consensus_queue.cc:260] T 71660575d94748d8a9329f909a42a0cd P 938a24b0e6a040b7966225af26558ff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } }
I20240501 14:05:35.213428 8680 tablet_copy_client.cc:523] T 71660575d94748d8a9329f909a42a0cd P 6dac90f01e744f9c8701a404440a4c77: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:05:35.214742 8676 ts_tablet_manager.cc:902] T e7d5f02ccba34254a4d20951878b1214 P 6dac90f01e744f9c8701a404440a4c77: Initiating tablet copy from peer be56773105e744919601127a94e98adf (127.29.124.132:35459)
I20240501 14:05:35.215444 8227 ts_tablet_manager.cc:1398] T 71660575d94748d8a9329f909a42a0cd P 938a24b0e6a040b7966225af26558ff8: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240501 14:05:35.216702 5276 tablet_copy_service.cc:342] P 1c3fdc5272484ac09ef3e5e36a08f27c: Request end of tablet copy session 938a24b0e6a040b7966225af26558ff8-71660575d94748d8a9329f909a42a0cd received from {username='slave'} at 127.29.124.133:49097
I20240501 14:05:35.216883 5276 tablet_copy_service.cc:433] P 1c3fdc5272484ac09ef3e5e36a08f27c: ending tablet copy session 938a24b0e6a040b7966225af26558ff8-71660575d94748d8a9329f909a42a0cd on tablet 71660575d94748d8a9329f909a42a0cd with peer 938a24b0e6a040b7966225af26558ff8
I20240501 14:05:35.219467 8409 ts_tablet_manager.cc:902] T e7d5f02ccba34254a4d20951878b1214 P 938a24b0e6a040b7966225af26558ff8: Initiating tablet copy from peer be56773105e744919601127a94e98adf (127.29.124.132:35459)
I20240501 14:05:35.222194 6892 raft_consensus.cc:1203] T 3d6859d53fd747a5a5e94722626fa522 P 938a24b0e6a040b7966225af26558ff8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:05:35.224772 5384 consensus_queue.cc:237] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } }
I20240501 14:05:35.227973 6892 raft_consensus.cc:1261] T 2c39b53867e047faaf24cc6f87a8100c P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:05:35.228614 8612 consensus_queue.cc:1035] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:05:35.231048 8612 raft_consensus.cc:2932] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:35.232843 8685 tablet_copy_client.cc:308] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 4360b59e3a154688a79ca5f9c0a37bc9: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.131:33209
I20240501 14:05:35.233557 5276 tablet_copy_service.cc:140] P 1c3fdc5272484ac09ef3e5e36a08f27c: Received BeginTabletCopySession request for tablet ec1e81011c6a46bc8a39aecf3a5e71e4 from peer 4360b59e3a154688a79ca5f9c0a37bc9 ({username='slave'} at 127.29.124.135:51325)
I20240501 14:05:35.233763 5276 tablet_copy_service.cc:161] P 1c3fdc5272484ac09ef3e5e36a08f27c: Beginning new tablet copy session on tablet ec1e81011c6a46bc8a39aecf3a5e71e4 from peer 4360b59e3a154688a79ca5f9c0a37bc9 at {username='slave'} at 127.29.124.135:51325: session id = 4360b59e3a154688a79ca5f9c0a37bc9-ec1e81011c6a46bc8a39aecf3a5e71e4
I20240501 14:05:35.233893 6892 raft_consensus.cc:2932] T 2c39b53867e047faaf24cc6f87a8100c P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:35.236758 5276 tablet_copy_source_session.cc:205] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:05:35.238649 8685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec1e81011c6a46bc8a39aecf3a5e71e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:35.239374 4847 catalog_manager.cc:5484] T 2c39b53867e047faaf24cc6f87a8100c P 938a24b0e6a040b7966225af26558ff8 reported cstate change: config changed from index 8 to 9, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:35.248437 5125 raft_consensus.cc:1261] T 2c39b53867e047faaf24cc6f87a8100c P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:05:35.249439 8612 consensus_queue.cc:1035] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:05:35.251289 5125 raft_consensus.cc:2932] T 2c39b53867e047faaf24cc6f87a8100c P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:35.253078 8680 tablet_bootstrap.cc:492] T 71660575d94748d8a9329f909a42a0cd P 6dac90f01e744f9c8701a404440a4c77: Bootstrap starting.
I20240501 14:05:35.256239 8685 tablet_copy_client.cc:783] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 4360b59e3a154688a79ca5f9c0a37bc9: tablet copy: Starting download of 0 data blocks...
I20240501 14:05:35.256538 8685 tablet_copy_client.cc:647] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 4360b59e3a154688a79ca5f9c0a37bc9: tablet copy: Starting download of 1 WAL segments...
I20240501 14:05:35.261480 8685 tablet_copy_client.cc:523] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 4360b59e3a154688a79ca5f9c0a37bc9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240501 14:05:35.267308 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2c39b53867e047faaf24cc6f87a8100c. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:35.272711 8685 tablet_bootstrap.cc:492] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 4360b59e3a154688a79ca5f9c0a37bc9: Bootstrap starting.
I20240501 14:05:35.287731 8590 tablet_replica.cc:330] T e7d5f02ccba34254a4d20951878b1214 P 19a6d2d8a08142d5bffcde3a37234692: stopping tablet replica
I20240501 14:05:35.288249 8590 raft_consensus.cc:2229] T e7d5f02ccba34254a4d20951878b1214 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 LEARNER]: Raft consensus shutting down.
I20240501 14:05:35.288522 8590 pending_rounds.cc:62] T e7d5f02ccba34254a4d20951878b1214 P 19a6d2d8a08142d5bffcde3a37234692: Trying to abort 1 pending ops.
I20240501 14:05:35.288616 8590 pending_rounds.cc:68] T e7d5f02ccba34254a4d20951878b1214 P 19a6d2d8a08142d5bffcde3a37234692: Aborting op as it isn't in flight: id { term: 1 index: 6 } timestamp: 7022888282825150464 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "e7d5f02ccba34254a4d20951878b1214" old_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } } new_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } } }
I20240501 14:05:35.289089 8590 raft_consensus.cc:2894] T e7d5f02ccba34254a4d20951878b1214 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 LEARNER]: Aborting config change with OpId 1.6: Aborted: Op aborted
I20240501 14:05:35.289220 8590 raft_consensus.cc:2866] T e7d5f02ccba34254a4d20951878b1214 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 LEARNER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 14:05:35.289613 8590 raft_consensus.cc:2258] T e7d5f02ccba34254a4d20951878b1214 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 LEARNER]: Raft consensus is shut down!
I20240501 14:05:35.291152 8590 ts_tablet_manager.cc:1875] T e7d5f02ccba34254a4d20951878b1214 P 19a6d2d8a08142d5bffcde3a37234692: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:05:35.300004 8557 tablet_bootstrap.cc:492] T ba6cba753e934e719c730593c0706346 P 4360b59e3a154688a79ca5f9c0a37bc9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:05:35.300473 8557 tablet_bootstrap.cc:492] T ba6cba753e934e719c730593c0706346 P 4360b59e3a154688a79ca5f9c0a37bc9: Bootstrap complete.
I20240501 14:05:35.303525 8500 raft_consensus.cc:1050] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c: attempting to promote NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 to VOTER
I20240501 14:05:35.304303 8500 consensus_queue.cc:237] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } }
I20240501 14:05:35.300734 8557 ts_tablet_manager.cc:1367] T ba6cba753e934e719c730593c0706346 P 4360b59e3a154688a79ca5f9c0a37bc9: Time spent bootstrapping tablet: real 0.230s user 0.050s sys 0.003s
I20240501 14:05:35.307672 8557 raft_consensus.cc:348] T ba6cba753e934e719c730593c0706346 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } }
I20240501 14:05:35.308003 8557 raft_consensus.cc:729] T ba6cba753e934e719c730593c0706346 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4360b59e3a154688a79ca5f9c0a37bc9, State: Initialized, Role: LEARNER
I20240501 14:05:35.308255 8557 consensus_queue.cc:260] T ba6cba753e934e719c730593c0706346 P 4360b59e3a154688a79ca5f9c0a37bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } }
I20240501 14:05:35.310581 8590 ts_tablet_manager.cc:1888] T e7d5f02ccba34254a4d20951878b1214 P 19a6d2d8a08142d5bffcde3a37234692: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 14:05:35.311161 8642 tablet_bootstrap.cc:492] T 51cbcf4888c440c2ad096c988ab2d968 P 938a24b0e6a040b7966225af26558ff8: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:05:35.311587 8642 tablet_bootstrap.cc:492] T 51cbcf4888c440c2ad096c988ab2d968 P 938a24b0e6a040b7966225af26558ff8: Bootstrap complete.
I20240501 14:05:35.311861 8642 ts_tablet_manager.cc:1367] T 51cbcf4888c440c2ad096c988ab2d968 P 938a24b0e6a040b7966225af26558ff8: Time spent bootstrapping tablet: real 0.178s user 0.044s sys 0.003s
I20240501 14:05:35.312677 8642 raft_consensus.cc:348] T 51cbcf4888c440c2ad096c988ab2d968 P 938a24b0e6a040b7966225af26558ff8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }
I20240501 14:05:35.312934 8642 raft_consensus.cc:729] T 51cbcf4888c440c2ad096c988ab2d968 P 938a24b0e6a040b7966225af26558ff8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 938a24b0e6a040b7966225af26558ff8, State: Initialized, Role: LEARNER
I20240501 14:05:35.318372 8557 ts_tablet_manager.cc:1398] T ba6cba753e934e719c730593c0706346 P 4360b59e3a154688a79ca5f9c0a37bc9: Time spent starting tablet: real 0.011s user 0.001s sys 0.002s
I20240501 14:05:35.319810 5015 tablet_copy_service.cc:342] P 04088ee3bea440a7bd5ff772152b9410: Request end of tablet copy session 4360b59e3a154688a79ca5f9c0a37bc9-ba6cba753e934e719c730593c0706346 received from {username='slave'} at 127.29.124.135:54069
I20240501 14:05:35.319954 5015 tablet_copy_service.cc:433] P 04088ee3bea440a7bd5ff772152b9410: ending tablet copy session 4360b59e3a154688a79ca5f9c0a37bc9-ba6cba753e934e719c730593c0706346 on tablet ba6cba753e934e719c730593c0706346 with peer 4360b59e3a154688a79ca5f9c0a37bc9
I20240501 14:05:35.321606 8590 log.cc:1196] T e7d5f02ccba34254a4d20951878b1214 P 19a6d2d8a08142d5bffcde3a37234692: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-5/wal/wals/e7d5f02ccba34254a4d20951878b1214
I20240501 14:05:35.313131 8642 consensus_queue.cc:260] T 51cbcf4888c440c2ad096c988ab2d968 P 938a24b0e6a040b7966225af26558ff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }
I20240501 14:05:35.322144 8590 ts_tablet_manager.cc:902] T e7d5f02ccba34254a4d20951878b1214 P 19a6d2d8a08142d5bffcde3a37234692: Initiating tablet copy from peer be56773105e744919601127a94e98adf (127.29.124.132:35459)
I20240501 14:05:35.322995 8642 ts_tablet_manager.cc:1398] T 51cbcf4888c440c2ad096c988ab2d968 P 938a24b0e6a040b7966225af26558ff8: Time spent starting tablet: real 0.011s user 0.002s sys 0.003s
I20240501 14:05:35.323737 5276 tablet_copy_service.cc:342] P 1c3fdc5272484ac09ef3e5e36a08f27c: Request end of tablet copy session 938a24b0e6a040b7966225af26558ff8-51cbcf4888c440c2ad096c988ab2d968 received from {username='slave'} at 127.29.124.133:49097
I20240501 14:05:35.323875 5276 tablet_copy_service.cc:433] P 1c3fdc5272484ac09ef3e5e36a08f27c: ending tablet copy session 938a24b0e6a040b7966225af26558ff8-51cbcf4888c440c2ad096c988ab2d968 on tablet 51cbcf4888c440c2ad096c988ab2d968 with peer 938a24b0e6a040b7966225af26558ff8
I20240501 14:05:35.330039 8594 tablet_bootstrap.cc:492] T 61a274b24f53458f88c2cde4f8dbe4f4 P 6dac90f01e744f9c8701a404440a4c77: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:05:35.331000 8594 tablet_bootstrap.cc:492] T 61a274b24f53458f88c2cde4f8dbe4f4 P 6dac90f01e744f9c8701a404440a4c77: Bootstrap complete.
I20240501 14:05:35.331290 8594 ts_tablet_manager.cc:1367] T 61a274b24f53458f88c2cde4f8dbe4f4 P 6dac90f01e744f9c8701a404440a4c77: Time spent bootstrapping tablet: real 0.183s user 0.044s sys 0.006s
I20240501 14:05:35.332136 8409 tablet_copy_client.cc:308] T e7d5f02ccba34254a4d20951878b1214 P 938a24b0e6a040b7966225af26558ff8: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.132:35459
I20240501 14:05:35.332077 8594 raft_consensus.cc:348] T 61a274b24f53458f88c2cde4f8dbe4f4 P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }
I20240501 14:05:35.332736 5406 tablet_copy_service.cc:140] P be56773105e744919601127a94e98adf: Received BeginTabletCopySession request for tablet e7d5f02ccba34254a4d20951878b1214 from peer 938a24b0e6a040b7966225af26558ff8 ({username='slave'} at 127.29.124.133:50583)
I20240501 14:05:35.332906 5406 tablet_copy_service.cc:161] P be56773105e744919601127a94e98adf: Beginning new tablet copy session on tablet e7d5f02ccba34254a4d20951878b1214 from peer 938a24b0e6a040b7966225af26558ff8 at {username='slave'} at 127.29.124.133:50583: session id = 938a24b0e6a040b7966225af26558ff8-e7d5f02ccba34254a4d20951878b1214
I20240501 14:05:35.335861 5406 tablet_copy_source_session.cc:205] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:05:35.336488 5125 raft_consensus.cc:1261] T 23ca2bf1e59f44fcac8a48aff8d056b1 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:35.336608 7025 raft_consensus.cc:1261] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 LEARNER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:35.337086 8409 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7d5f02ccba34254a4d20951878b1214. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:35.332403 8594 raft_consensus.cc:729] T 61a274b24f53458f88c2cde4f8dbe4f4 P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6dac90f01e744f9c8701a404440a4c77, State: Initialized, Role: LEARNER
I20240501 14:05:35.337388 5382 raft_consensus.cc:1261] T 23ca2bf1e59f44fcac8a48aff8d056b1 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:35.337337 8659 consensus_queue.cc:1035] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:35.337694 8594 consensus_queue.cc:260] T 61a274b24f53458f88c2cde4f8dbe4f4 P 6dac90f01e744f9c8701a404440a4c77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }
I20240501 14:05:35.340343 8277 consensus_queue.cc:1035] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:35.341518 8277 consensus_queue.cc:1035] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:35.350905 8590 tablet_copy_client.cc:308] T e7d5f02ccba34254a4d20951878b1214 P 19a6d2d8a08142d5bffcde3a37234692: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.132:35459
I20240501 14:05:35.352224 5406 tablet_copy_service.cc:140] P be56773105e744919601127a94e98adf: Received BeginTabletCopySession request for tablet e7d5f02ccba34254a4d20951878b1214 from peer 19a6d2d8a08142d5bffcde3a37234692 ({username='slave'} at 127.29.124.134:56125)
I20240501 14:05:35.352440 5406 tablet_copy_service.cc:161] P be56773105e744919601127a94e98adf: Beginning new tablet copy session on tablet e7d5f02ccba34254a4d20951878b1214 from peer 19a6d2d8a08142d5bffcde3a37234692 at {username='slave'} at 127.29.124.134:56125: session id = 19a6d2d8a08142d5bffcde3a37234692-e7d5f02ccba34254a4d20951878b1214
I20240501 14:05:35.355520 5406 tablet_copy_source_session.cc:205] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:05:35.356828 8590 ts_tablet_manager.cc:1875] T e7d5f02ccba34254a4d20951878b1214 P 19a6d2d8a08142d5bffcde3a37234692: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 14:05:35.357789 8241 tablet_bootstrap.cc:492] T 51cbcf4888c440c2ad096c988ab2d968 P 4360b59e3a154688a79ca5f9c0a37bc9: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:05:35.358244 8241 tablet_bootstrap.cc:492] T 51cbcf4888c440c2ad096c988ab2d968 P 4360b59e3a154688a79ca5f9c0a37bc9: Bootstrap complete.
I20240501 14:05:35.358527 8241 ts_tablet_manager.cc:1367] T 51cbcf4888c440c2ad096c988ab2d968 P 4360b59e3a154688a79ca5f9c0a37bc9: Time spent bootstrapping tablet: real 0.295s user 0.043s sys 0.011s
I20240501 14:05:35.359316 8241 raft_consensus.cc:348] T 51cbcf4888c440c2ad096c988ab2d968 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }
I20240501 14:05:35.359603 8241 raft_consensus.cc:729] T 51cbcf4888c440c2ad096c988ab2d968 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4360b59e3a154688a79ca5f9c0a37bc9, State: Initialized, Role: LEARNER
I20240501 14:05:35.359822 8241 consensus_queue.cc:260] T 51cbcf4888c440c2ad096c988ab2d968 P 4360b59e3a154688a79ca5f9c0a37bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }
I20240501 14:05:35.360548 8409 tablet_copy_client.cc:783] T e7d5f02ccba34254a4d20951878b1214 P 938a24b0e6a040b7966225af26558ff8: tablet copy: Starting download of 0 data blocks...
I20240501 14:05:35.361788 8587 tablet_bootstrap.cc:492] T 61a274b24f53458f88c2cde4f8dbe4f4 P 19a6d2d8a08142d5bffcde3a37234692: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:05:35.362301 8587 tablet_bootstrap.cc:492] T 61a274b24f53458f88c2cde4f8dbe4f4 P 19a6d2d8a08142d5bffcde3a37234692: Bootstrap complete.
I20240501 14:05:35.362607 8587 ts_tablet_manager.cc:1367] T 61a274b24f53458f88c2cde4f8dbe4f4 P 19a6d2d8a08142d5bffcde3a37234692: Time spent bootstrapping tablet: real 0.278s user 0.055s sys 0.000s
I20240501 14:05:35.363500 8587 raft_consensus.cc:348] T 61a274b24f53458f88c2cde4f8dbe4f4 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } }
I20240501 14:05:35.363503 8500 raft_consensus.cc:2932] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:35.360821 8409 tablet_copy_client.cc:647] T e7d5f02ccba34254a4d20951878b1214 P 938a24b0e6a040b7966225af26558ff8: tablet copy: Starting download of 1 WAL segments...
I20240501 14:05:35.365494 8241 ts_tablet_manager.cc:1398] T 51cbcf4888c440c2ad096c988ab2d968 P 4360b59e3a154688a79ca5f9c0a37bc9: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240501 14:05:35.365790 8587 raft_consensus.cc:729] T 61a274b24f53458f88c2cde4f8dbe4f4 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 19a6d2d8a08142d5bffcde3a37234692, State: Initialized, Role: LEARNER
I20240501 14:05:35.366176 5276 tablet_copy_service.cc:342] P 1c3fdc5272484ac09ef3e5e36a08f27c: Request end of tablet copy session 4360b59e3a154688a79ca5f9c0a37bc9-51cbcf4888c440c2ad096c988ab2d968 received from {username='slave'} at 127.29.124.135:51325
I20240501 14:05:35.366062 8587 consensus_queue.cc:260] T 61a274b24f53458f88c2cde4f8dbe4f4 P 19a6d2d8a08142d5bffcde3a37234692 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } }
I20240501 14:05:35.366314 5276 tablet_copy_service.cc:433] P 1c3fdc5272484ac09ef3e5e36a08f27c: ending tablet copy session 4360b59e3a154688a79ca5f9c0a37bc9-51cbcf4888c440c2ad096c988ab2d968 on tablet 51cbcf4888c440c2ad096c988ab2d968 with peer 4360b59e3a154688a79ca5f9c0a37bc9
I20240501 14:05:35.369598 8587 ts_tablet_manager.cc:1398] T 61a274b24f53458f88c2cde4f8dbe4f4 P 19a6d2d8a08142d5bffcde3a37234692: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240501 14:05:35.369462 5384 raft_consensus.cc:2932] T 23ca2bf1e59f44fcac8a48aff8d056b1 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:35.370801 5405 tablet_copy_service.cc:342] P be56773105e744919601127a94e98adf: Request end of tablet copy session 19a6d2d8a08142d5bffcde3a37234692-61a274b24f53458f88c2cde4f8dbe4f4 received from {username='slave'} at 127.29.124.134:56125
I20240501 14:05:35.370940 5405 tablet_copy_service.cc:433] P be56773105e744919601127a94e98adf: ending tablet copy session 19a6d2d8a08142d5bffcde3a37234692-61a274b24f53458f88c2cde4f8dbe4f4 on tablet 61a274b24f53458f88c2cde4f8dbe4f4 with peer 19a6d2d8a08142d5bffcde3a37234692
I20240501 14:05:35.373950 8590 ts_tablet_manager.cc:1888] T e7d5f02ccba34254a4d20951878b1214 P 19a6d2d8a08142d5bffcde3a37234692: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240501 14:05:35.374606 8409 tablet_copy_client.cc:523] T e7d5f02ccba34254a4d20951878b1214 P 938a24b0e6a040b7966225af26558ff8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:05:35.375941 4853 catalog_manager.cc:5484] T 23ca2bf1e59f44fcac8a48aff8d056b1 P be56773105e744919601127a94e98adf reported cstate change: config changed from index 4 to 5, 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:35.377130 5125 raft_consensus.cc:2932] T 23ca2bf1e59f44fcac8a48aff8d056b1 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:35.378814 8409 tablet_bootstrap.cc:492] T e7d5f02ccba34254a4d20951878b1214 P 938a24b0e6a040b7966225af26558ff8: Bootstrap starting.
I20240501 14:05:35.385229 8590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7d5f02ccba34254a4d20951878b1214. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:35.384191 8587 ts_tablet_manager.cc:902] T 404e2851055541f0be1bb0971a7cc72a P 19a6d2d8a08142d5bffcde3a37234692: Initiating tablet copy from peer 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209)
I20240501 14:05:35.389750 8594 ts_tablet_manager.cc:1398] T 61a274b24f53458f88c2cde4f8dbe4f4 P 6dac90f01e744f9c8701a404440a4c77: Time spent starting tablet: real 0.057s user 0.003s sys 0.001s
I20240501 14:05:35.394577 5406 tablet_copy_service.cc:342] P be56773105e744919601127a94e98adf: Request end of tablet copy session 6dac90f01e744f9c8701a404440a4c77-61a274b24f53458f88c2cde4f8dbe4f4 received from {username='slave'} at 127.29.124.136:50719
I20240501 14:05:35.394731 5406 tablet_copy_service.cc:433] P be56773105e744919601127a94e98adf: ending tablet copy session 6dac90f01e744f9c8701a404440a4c77-61a274b24f53458f88c2cde4f8dbe4f4 on tablet 61a274b24f53458f88c2cde4f8dbe4f4 with peer 6dac90f01e744f9c8701a404440a4c77
I20240501 14:05:35.396430 7025 raft_consensus.cc:2932] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:35.398437 5256 consensus_queue.cc:237] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } }
I20240501 14:05:35.401350 5124 raft_consensus.cc:1261] T 23ca2bf1e59f44fcac8a48aff8d056b1 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:35.402422 8277 consensus_queue.cc:1035] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:35.403105 8590 tablet_copy_client.cc:783] T e7d5f02ccba34254a4d20951878b1214 P 19a6d2d8a08142d5bffcde3a37234692: tablet copy: Starting download of 0 data blocks...
I20240501 14:05:35.403702 8590 tablet_copy_client.cc:647] T e7d5f02ccba34254a4d20951878b1214 P 19a6d2d8a08142d5bffcde3a37234692: tablet copy: Starting download of 1 WAL segments...
I20240501 14:05:35.404983 8659 raft_consensus.cc:2932] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER be56773105e744919601127a94e98adf (127.29.124.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:35.405288 5126 raft_consensus.cc:2932] T 23ca2bf1e59f44fcac8a48aff8d056b1 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER be56773105e744919601127a94e98adf (127.29.124.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:35.409219 4838 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 23ca2bf1e59f44fcac8a48aff8d056b1 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:05:35.414113 4854 catalog_manager.cc:5484] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c reported cstate change: config changed from index 5 to 6, VOTER be56773105e744919601127a94e98adf (127.29.124.132) evicted. New cstate: current_term: 1 leader_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:05:35.419086 5365 tablet_service.cc:1514] Processing DeleteTablet for tablet 23ca2bf1e59f44fcac8a48aff8d056b1 with delete_type TABLET_DATA_TOMBSTONED (TS be56773105e744919601127a94e98adf not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:43020
I20240501 14:05:35.422863 8736 tablet_replica.cc:330] T 23ca2bf1e59f44fcac8a48aff8d056b1 P be56773105e744919601127a94e98adf: stopping tablet replica
I20240501 14:05:35.424183 8736 raft_consensus.cc:2229] T 23ca2bf1e59f44fcac8a48aff8d056b1 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:05:35.425057 8736 raft_consensus.cc:2258] T 23ca2bf1e59f44fcac8a48aff8d056b1 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:05:35.424831 5125 consensus_queue.cc:237] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } }
I20240501 14:05:35.427506 8736 ts_tablet_manager.cc:1875] T 23ca2bf1e59f44fcac8a48aff8d056b1 P be56773105e744919601127a94e98adf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:05:35.432044 8676 tablet_copy_client.cc:308] T e7d5f02ccba34254a4d20951878b1214 P 6dac90f01e744f9c8701a404440a4c77: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.132:35459
I20240501 14:05:35.432722 5406 tablet_copy_service.cc:140] P be56773105e744919601127a94e98adf: Received BeginTabletCopySession request for tablet e7d5f02ccba34254a4d20951878b1214 from peer 6dac90f01e744f9c8701a404440a4c77 ({username='slave'} at 127.29.124.136:50719)
I20240501 14:05:35.432881 5406 tablet_copy_service.cc:161] P be56773105e744919601127a94e98adf: Beginning new tablet copy session on tablet e7d5f02ccba34254a4d20951878b1214 from peer 6dac90f01e744f9c8701a404440a4c77 at {username='slave'} at 127.29.124.136:50719: session id = 6dac90f01e744f9c8701a404440a4c77-e7d5f02ccba34254a4d20951878b1214
I20240501 14:05:35.436054 5406 tablet_copy_source_session.cc:205] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:05:35.436717 8590 tablet_copy_client.cc:523] T e7d5f02ccba34254a4d20951878b1214 P 19a6d2d8a08142d5bffcde3a37234692: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:05:35.445148 8676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7d5f02ccba34254a4d20951878b1214. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:35.446352 8590 tablet_bootstrap.cc:492] T e7d5f02ccba34254a4d20951878b1214 P 19a6d2d8a08142d5bffcde3a37234692: Bootstrap starting.
I20240501 14:05:35.459683 7024 raft_consensus.cc:1261] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:35.460204 4992 raft_consensus.cc:1261] T 43b46fcf827c4275a47291afb107c290 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:05:35.460788 8279 consensus_queue.cc:1035] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:05:35.461059 5382 raft_consensus.cc:1261] T 43b46fcf827c4275a47291afb107c290 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:05:35.461771 8659 consensus_queue.cc:1035] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:35.464269 8279 consensus_queue.cc:1035] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:05:35.466740 7024 raft_consensus.cc:2932] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER be56773105e744919601127a94e98adf (127.29.124.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:35.467433 8736 ts_tablet_manager.cc:1888] T 23ca2bf1e59f44fcac8a48aff8d056b1 P be56773105e744919601127a94e98adf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:05:35.467566 8736 log.cc:1196] T 23ca2bf1e59f44fcac8a48aff8d056b1 P be56773105e744919601127a94e98adf: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/wal/wals/23ca2bf1e59f44fcac8a48aff8d056b1
I20240501 14:05:35.470104 4839 catalog_manager.cc:4830] TS be56773105e744919601127a94e98adf (127.29.124.132:35459): tablet 23ca2bf1e59f44fcac8a48aff8d056b1 (table rebalance_test_table_7 [id=763c884aecd640b49c65c692fbde9422]) successfully deleted
I20240501 14:05:35.472410 6892 raft_consensus.cc:1203] T 51cbcf4888c440c2ad096c988ab2d968 P 938a24b0e6a040b7966225af26558ff8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
W20240501 14:05:35.474717 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 43b46fcf827c4275a47291afb107c290. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:35.478696 8279 raft_consensus.cc:2932] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } }
I20240501 14:05:35.482143 4993 raft_consensus.cc:2932] T 43b46fcf827c4275a47291afb107c290 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } }
I20240501 14:05:35.485503 4854 catalog_manager.cc:5484] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 reported cstate change: config changed from index -1 to 4, NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) added. New cstate: current_term: 1 leader_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:05:35.491266 7155 raft_consensus.cc:1203] T ba6cba753e934e719c730593c0706346 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:05:35.498440 6892 raft_consensus.cc:1203] T 71660575d94748d8a9329f909a42a0cd P 938a24b0e6a040b7966225af26558ff8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:05:35.511960 8594 tablet_replica.cc:330] T 61a274b24f53458f88c2cde4f8dbe4f4 P 6dac90f01e744f9c8701a404440a4c77: stopping tablet replica
I20240501 14:05:35.512523 8594 raft_consensus.cc:2229] T 61a274b24f53458f88c2cde4f8dbe4f4 P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Raft consensus shutting down.
I20240501 14:05:35.512825 8594 raft_consensus.cc:2258] T 61a274b24f53458f88c2cde4f8dbe4f4 P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Raft consensus is shut down!
I20240501 14:05:35.514364 8594 ts_tablet_manager.cc:1875] T 61a274b24f53458f88c2cde4f8dbe4f4 P 6dac90f01e744f9c8701a404440a4c77: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:05:35.521034 8594 ts_tablet_manager.cc:1888] T 61a274b24f53458f88c2cde4f8dbe4f4 P 6dac90f01e744f9c8701a404440a4c77: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240501 14:05:35.521373 8594 log.cc:1196] T 61a274b24f53458f88c2cde4f8dbe4f4 P 6dac90f01e744f9c8701a404440a4c77: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-7/wal/wals/61a274b24f53458f88c2cde4f8dbe4f4
I20240501 14:05:35.528996 5384 raft_consensus.cc:2932] T 43b46fcf827c4275a47291afb107c290 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } }
I20240501 14:05:35.532377 8594 ts_tablet_manager.cc:902] T 61a274b24f53458f88c2cde4f8dbe4f4 P 6dac90f01e744f9c8701a404440a4c77: Initiating tablet copy from peer be56773105e744919601127a94e98adf (127.29.124.132:35459)
I20240501 14:05:35.543269 8685 tablet_bootstrap.cc:492] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 4360b59e3a154688a79ca5f9c0a37bc9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:05:35.545837 8685 tablet_bootstrap.cc:492] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 4360b59e3a154688a79ca5f9c0a37bc9: Bootstrap complete.
I20240501 14:05:35.546406 8594 tablet_copy_client.cc:308] T 61a274b24f53458f88c2cde4f8dbe4f4 P 6dac90f01e744f9c8701a404440a4c77: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.132:35459
I20240501 14:05:35.546942 8685 ts_tablet_manager.cc:1367] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 4360b59e3a154688a79ca5f9c0a37bc9: Time spent bootstrapping tablet: real 0.277s user 0.047s sys 0.010s
I20240501 14:05:35.547870 5406 tablet_copy_service.cc:140] P be56773105e744919601127a94e98adf: Received BeginTabletCopySession request for tablet 61a274b24f53458f88c2cde4f8dbe4f4 from peer 6dac90f01e744f9c8701a404440a4c77 ({username='slave'} at 127.29.124.136:50719)
I20240501 14:05:35.548048 5406 tablet_copy_service.cc:161] P be56773105e744919601127a94e98adf: Beginning new tablet copy session on tablet 61a274b24f53458f88c2cde4f8dbe4f4 from peer 6dac90f01e744f9c8701a404440a4c77 at {username='slave'} at 127.29.124.136:50719: session id = 6dac90f01e744f9c8701a404440a4c77-61a274b24f53458f88c2cde4f8dbe4f4
I20240501 14:05:35.550894 5406 tablet_copy_source_session.cc:205] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:05:35.551906 8594 ts_tablet_manager.cc:1875] T 61a274b24f53458f88c2cde4f8dbe4f4 P 6dac90f01e744f9c8701a404440a4c77: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 14:05:35.552680 8685 raft_consensus.cc:348] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } }
I20240501 14:05:35.555907 8685 raft_consensus.cc:729] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4360b59e3a154688a79ca5f9c0a37bc9, State: Initialized, Role: LEARNER
I20240501 14:05:35.556234 8685 consensus_queue.cc:260] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 4360b59e3a154688a79ca5f9c0a37bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } }
I20240501 14:05:35.558236 8685 ts_tablet_manager.cc:1398] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 4360b59e3a154688a79ca5f9c0a37bc9: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240501 14:05:35.560977 8587 tablet_copy_client.cc:308] T 404e2851055541f0be1bb0971a7cc72a P 19a6d2d8a08142d5bffcde3a37234692: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.131:33209
I20240501 14:05:35.561722 5276 tablet_copy_service.cc:140] P 1c3fdc5272484ac09ef3e5e36a08f27c: Received BeginTabletCopySession request for tablet 404e2851055541f0be1bb0971a7cc72a from peer 19a6d2d8a08142d5bffcde3a37234692 ({username='slave'} at 127.29.124.134:43435)
I20240501 14:05:35.561889 5276 tablet_copy_service.cc:161] P 1c3fdc5272484ac09ef3e5e36a08f27c: Beginning new tablet copy session on tablet 404e2851055541f0be1bb0971a7cc72a from peer 19a6d2d8a08142d5bffcde3a37234692 at {username='slave'} at 127.29.124.134:43435: session id = 19a6d2d8a08142d5bffcde3a37234692-404e2851055541f0be1bb0971a7cc72a
I20240501 14:05:35.564548 5276 tablet_copy_source_session.cc:205] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:05:35.565752 8587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 404e2851055541f0be1bb0971a7cc72a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:35.569379 5276 tablet_copy_service.cc:342] P 1c3fdc5272484ac09ef3e5e36a08f27c: Request end of tablet copy session 4360b59e3a154688a79ca5f9c0a37bc9-ec1e81011c6a46bc8a39aecf3a5e71e4 received from {username='slave'} at 127.29.124.135:51325
I20240501 14:05:35.569514 5276 tablet_copy_service.cc:433] P 1c3fdc5272484ac09ef3e5e36a08f27c: ending tablet copy session 4360b59e3a154688a79ca5f9c0a37bc9-ec1e81011c6a46bc8a39aecf3a5e71e4 on tablet ec1e81011c6a46bc8a39aecf3a5e71e4 with peer 4360b59e3a154688a79ca5f9c0a37bc9
I20240501 14:05:35.571251 8680 tablet_bootstrap.cc:492] T 71660575d94748d8a9329f909a42a0cd P 6dac90f01e744f9c8701a404440a4c77: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:05:35.572342 8642 ts_tablet_manager.cc:902] T 61a274b24f53458f88c2cde4f8dbe4f4 P 938a24b0e6a040b7966225af26558ff8: Initiating tablet copy from peer be56773105e744919601127a94e98adf (127.29.124.132:35459)
I20240501 14:05:35.575639 8587 tablet_copy_client.cc:783] T 404e2851055541f0be1bb0971a7cc72a P 19a6d2d8a08142d5bffcde3a37234692: tablet copy: Starting download of 0 data blocks...
I20240501 14:05:35.575886 8587 tablet_copy_client.cc:647] T 404e2851055541f0be1bb0971a7cc72a P 19a6d2d8a08142d5bffcde3a37234692: tablet copy: Starting download of 1 WAL segments...
I20240501 14:05:35.577575 8587 tablet_copy_client.cc:523] T 404e2851055541f0be1bb0971a7cc72a P 19a6d2d8a08142d5bffcde3a37234692: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:05:35.581164 8587 tablet_bootstrap.cc:492] T 404e2851055541f0be1bb0971a7cc72a P 19a6d2d8a08142d5bffcde3a37234692: Bootstrap starting.
I20240501 14:05:35.584585 8409 tablet_bootstrap.cc:492] T e7d5f02ccba34254a4d20951878b1214 P 938a24b0e6a040b7966225af26558ff8: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:05:35.585024 8409 tablet_bootstrap.cc:492] T e7d5f02ccba34254a4d20951878b1214 P 938a24b0e6a040b7966225af26558ff8: Bootstrap complete.
I20240501 14:05:35.573717 8676 tablet_copy_client.cc:783] T e7d5f02ccba34254a4d20951878b1214 P 6dac90f01e744f9c8701a404440a4c77: tablet copy: Starting download of 0 data blocks...
I20240501 14:05:35.589883 8676 tablet_copy_client.cc:647] T e7d5f02ccba34254a4d20951878b1214 P 6dac90f01e744f9c8701a404440a4c77: tablet copy: Starting download of 1 WAL segments...
I20240501 14:05:35.592352 8676 tablet_copy_client.cc:523] T e7d5f02ccba34254a4d20951878b1214 P 6dac90f01e744f9c8701a404440a4c77: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:05:35.594864 8590 tablet_bootstrap.cc:492] T e7d5f02ccba34254a4d20951878b1214 P 19a6d2d8a08142d5bffcde3a37234692: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:05:35.595264 8590 tablet_bootstrap.cc:492] T e7d5f02ccba34254a4d20951878b1214 P 19a6d2d8a08142d5bffcde3a37234692: Bootstrap complete.
I20240501 14:05:35.595525 8590 ts_tablet_manager.cc:1367] T e7d5f02ccba34254a4d20951878b1214 P 19a6d2d8a08142d5bffcde3a37234692: Time spent bootstrapping tablet: real 0.149s user 0.047s sys 0.003s
I20240501 14:05:35.595746 8409 ts_tablet_manager.cc:1367] T e7d5f02ccba34254a4d20951878b1214 P 938a24b0e6a040b7966225af26558ff8: Time spent bootstrapping tablet: real 0.217s user 0.050s sys 0.005s
I20240501 14:05:35.596364 8680 tablet_bootstrap.cc:492] T 71660575d94748d8a9329f909a42a0cd P 6dac90f01e744f9c8701a404440a4c77: Bootstrap complete.
I20240501 14:05:35.596341 8590 raft_consensus.cc:348] T e7d5f02ccba34254a4d20951878b1214 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }
I20240501 14:05:35.596654 8590 raft_consensus.cc:729] T e7d5f02ccba34254a4d20951878b1214 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 19a6d2d8a08142d5bffcde3a37234692, State: Initialized, Role: LEARNER
I20240501 14:05:35.596902 8590 consensus_queue.cc:260] T e7d5f02ccba34254a4d20951878b1214 P 19a6d2d8a08142d5bffcde3a37234692 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }
I20240501 14:05:35.597620 8590 ts_tablet_manager.cc:1398] T e7d5f02ccba34254a4d20951878b1214 P 19a6d2d8a08142d5bffcde3a37234692: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:05:35.597955 8680 ts_tablet_manager.cc:1367] T 71660575d94748d8a9329f909a42a0cd P 6dac90f01e744f9c8701a404440a4c77: Time spent bootstrapping tablet: real 0.345s user 0.040s sys 0.008s
I20240501 14:05:35.598322 5406 tablet_copy_service.cc:342] P be56773105e744919601127a94e98adf: Request end of tablet copy session 19a6d2d8a08142d5bffcde3a37234692-e7d5f02ccba34254a4d20951878b1214 received from {username='slave'} at 127.29.124.134:56125
I20240501 14:05:35.598469 5406 tablet_copy_service.cc:433] P be56773105e744919601127a94e98adf: ending tablet copy session 19a6d2d8a08142d5bffcde3a37234692-e7d5f02ccba34254a4d20951878b1214 on tablet e7d5f02ccba34254a4d20951878b1214 with peer 19a6d2d8a08142d5bffcde3a37234692
I20240501 14:05:35.600085 8680 raft_consensus.cc:348] T 71660575d94748d8a9329f909a42a0cd P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } }
I20240501 14:05:35.603916 8680 raft_consensus.cc:729] T 71660575d94748d8a9329f909a42a0cd P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6dac90f01e744f9c8701a404440a4c77, State: Initialized, Role: LEARNER
I20240501 14:05:35.604260 8680 consensus_queue.cc:260] T 71660575d94748d8a9329f909a42a0cd P 6dac90f01e744f9c8701a404440a4c77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } }
I20240501 14:05:35.600912 8676 tablet_bootstrap.cc:492] T e7d5f02ccba34254a4d20951878b1214 P 6dac90f01e744f9c8701a404440a4c77: Bootstrap starting.
I20240501 14:05:35.607039 8680 ts_tablet_manager.cc:1398] T 71660575d94748d8a9329f909a42a0cd P 6dac90f01e744f9c8701a404440a4c77: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240501 14:05:35.607888 5276 tablet_copy_service.cc:342] P 1c3fdc5272484ac09ef3e5e36a08f27c: Request end of tablet copy session 6dac90f01e744f9c8701a404440a4c77-71660575d94748d8a9329f909a42a0cd received from {username='slave'} at 127.29.124.136:49141
I20240501 14:05:35.608022 5276 tablet_copy_service.cc:433] P 1c3fdc5272484ac09ef3e5e36a08f27c: ending tablet copy session 6dac90f01e744f9c8701a404440a4c77-71660575d94748d8a9329f909a42a0cd on tablet 71660575d94748d8a9329f909a42a0cd with peer 6dac90f01e744f9c8701a404440a4c77
I20240501 14:05:35.589149 8594 ts_tablet_manager.cc:1888] T 61a274b24f53458f88c2cde4f8dbe4f4 P 6dac90f01e744f9c8701a404440a4c77: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.6
I20240501 14:05:35.611200 8594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61a274b24f53458f88c2cde4f8dbe4f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:35.623070 8594 tablet_copy_client.cc:783] T 61a274b24f53458f88c2cde4f8dbe4f4 P 6dac90f01e744f9c8701a404440a4c77: tablet copy: Starting download of 0 data blocks...
I20240501 14:05:35.623366 8594 tablet_copy_client.cc:647] T 61a274b24f53458f88c2cde4f8dbe4f4 P 6dac90f01e744f9c8701a404440a4c77: tablet copy: Starting download of 1 WAL segments...
I20240501 14:05:35.625989 8594 tablet_copy_client.cc:523] T 61a274b24f53458f88c2cde4f8dbe4f4 P 6dac90f01e744f9c8701a404440a4c77: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:05:35.627084 7024 raft_consensus.cc:2932] T 61a274b24f53458f88c2cde4f8dbe4f4 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:35.598639 8409 raft_consensus.cc:348] T e7d5f02ccba34254a4d20951878b1214 P 938a24b0e6a040b7966225af26558ff8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }
I20240501 14:05:35.634822 8409 raft_consensus.cc:729] T e7d5f02ccba34254a4d20951878b1214 P 938a24b0e6a040b7966225af26558ff8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 938a24b0e6a040b7966225af26558ff8, State: Initialized, Role: LEARNER
I20240501 14:05:35.635529 8409 consensus_queue.cc:260] T e7d5f02ccba34254a4d20951878b1214 P 938a24b0e6a040b7966225af26558ff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }
I20240501 14:05:35.637415 8409 ts_tablet_manager.cc:1398] T e7d5f02ccba34254a4d20951878b1214 P 938a24b0e6a040b7966225af26558ff8: Time spent starting tablet: real 0.042s user 0.002s sys 0.000s
I20240501 14:05:35.638370 5406 tablet_copy_service.cc:342] P be56773105e744919601127a94e98adf: Request end of tablet copy session 938a24b0e6a040b7966225af26558ff8-e7d5f02ccba34254a4d20951878b1214 received from {username='slave'} at 127.29.124.133:50583
I20240501 14:05:35.638520 5406 tablet_copy_service.cc:433] P be56773105e744919601127a94e98adf: ending tablet copy session 938a24b0e6a040b7966225af26558ff8-e7d5f02ccba34254a4d20951878b1214 on tablet e7d5f02ccba34254a4d20951878b1214 with peer 938a24b0e6a040b7966225af26558ff8
I20240501 14:05:35.647210 8594 tablet_bootstrap.cc:492] T 61a274b24f53458f88c2cde4f8dbe4f4 P 6dac90f01e744f9c8701a404440a4c77: Bootstrap starting.
I20240501 14:05:35.648115 8680 ts_tablet_manager.cc:902] T 2c39b53867e047faaf24cc6f87a8100c P 6dac90f01e744f9c8701a404440a4c77: Initiating tablet copy from peer be56773105e744919601127a94e98adf (127.29.124.132:35459)
I20240501 14:05:35.659933 6892 raft_consensus.cc:1203] T e7d5f02ccba34254a4d20951878b1214 P 938a24b0e6a040b7966225af26558ff8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240501 14:05:35.670048 5382 consensus_queue.cc:237] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } }
W20240501 14:05:35.674821 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 71bdb382b08341218c8bf2459fe65fc0. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:35.675819 4993 raft_consensus.cc:1261] T 71bdb382b08341218c8bf2459fe65fc0 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:05:35.676695 7155 raft_consensus.cc:1203] T 51cbcf4888c440c2ad096c988ab2d968 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:05:35.677791 8352 consensus_queue.cc:1035] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:05:35.678659 8642 tablet_copy_client.cc:308] T 61a274b24f53458f88c2cde4f8dbe4f4 P 938a24b0e6a040b7966225af26558ff8: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.132:35459
I20240501 14:05:35.679769 5406 tablet_copy_service.cc:140] P be56773105e744919601127a94e98adf: Received BeginTabletCopySession request for tablet 61a274b24f53458f88c2cde4f8dbe4f4 from peer 938a24b0e6a040b7966225af26558ff8 ({username='slave'} at 127.29.124.133:50583)
I20240501 14:05:35.679941 5406 tablet_copy_service.cc:161] P be56773105e744919601127a94e98adf: Beginning new tablet copy session on tablet 61a274b24f53458f88c2cde4f8dbe4f4 from peer 938a24b0e6a040b7966225af26558ff8 at {username='slave'} at 127.29.124.133:50583: session id = 938a24b0e6a040b7966225af26558ff8-61a274b24f53458f88c2cde4f8dbe4f4
I20240501 14:05:35.683465 5406 tablet_copy_source_session.cc:205] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:05:35.687505 8642 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61a274b24f53458f88c2cde4f8dbe4f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:35.687669 5256 raft_consensus.cc:1261] T 71bdb382b08341218c8bf2459fe65fc0 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:05:35.688596 8279 raft_consensus.cc:1050] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4: attempting to promote NON_VOTER 938a24b0e6a040b7966225af26558ff8 to VOTER
I20240501 14:05:35.690007 8279 consensus_queue.cc:237] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:05:35.691512 8352 consensus_queue.cc:1035] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:05:35.702005 8352 raft_consensus.cc:2932] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:35.706246 4992 raft_consensus.cc:2932] T 71bdb382b08341218c8bf2459fe65fc0 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:35.711016 8642 tablet_copy_client.cc:783] T 61a274b24f53458f88c2cde4f8dbe4f4 P 938a24b0e6a040b7966225af26558ff8: tablet copy: Starting download of 0 data blocks...
I20240501 14:05:35.711272 8642 tablet_copy_client.cc:647] T 61a274b24f53458f88c2cde4f8dbe4f4 P 938a24b0e6a040b7966225af26558ff8: tablet copy: Starting download of 1 WAL segments...
I20240501 14:05:35.711710 7155 raft_consensus.cc:1203] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:05:35.716084 8642 tablet_copy_client.cc:523] T 61a274b24f53458f88c2cde4f8dbe4f4 P 938a24b0e6a040b7966225af26558ff8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:05:35.717329 4846 catalog_manager.cc:5484] T 71bdb382b08341218c8bf2459fe65fc0 P 04088ee3bea440a7bd5ff772152b9410 reported cstate change: config changed from index -1 to 4, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) added. New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:35.726847 8642 tablet_bootstrap.cc:492] T 61a274b24f53458f88c2cde4f8dbe4f4 P 938a24b0e6a040b7966225af26558ff8: Bootstrap starting.
I20240501 14:05:35.746673 8680 tablet_copy_client.cc:308] T 2c39b53867e047faaf24cc6f87a8100c P 6dac90f01e744f9c8701a404440a4c77: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.132:35459
I20240501 14:05:35.747611 5406 tablet_copy_service.cc:140] P be56773105e744919601127a94e98adf: Received BeginTabletCopySession request for tablet 2c39b53867e047faaf24cc6f87a8100c from peer 6dac90f01e744f9c8701a404440a4c77 ({username='slave'} at 127.29.124.136:50719)
I20240501 14:05:35.747776 5406 tablet_copy_service.cc:161] P be56773105e744919601127a94e98adf: Beginning new tablet copy session on tablet 2c39b53867e047faaf24cc6f87a8100c from peer 6dac90f01e744f9c8701a404440a4c77 at {username='slave'} at 127.29.124.136:50719: session id = 6dac90f01e744f9c8701a404440a4c77-2c39b53867e047faaf24cc6f87a8100c
I20240501 14:05:35.750762 5406 tablet_copy_source_session.cc:205] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:05:35.752449 8680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c39b53867e047faaf24cc6f87a8100c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:35.759685 8680 tablet_copy_client.cc:783] T 2c39b53867e047faaf24cc6f87a8100c P 6dac90f01e744f9c8701a404440a4c77: tablet copy: Starting download of 0 data blocks...
I20240501 14:05:35.759919 8680 tablet_copy_client.cc:647] T 2c39b53867e047faaf24cc6f87a8100c P 6dac90f01e744f9c8701a404440a4c77: tablet copy: Starting download of 1 WAL segments...
I20240501 14:05:35.763160 6892 raft_consensus.cc:1261] T 3d6859d53fd747a5a5e94722626fa522 P 938a24b0e6a040b7966225af26558ff8 [term 1 LEARNER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:35.763751 8279 consensus_queue.cc:1035] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:35.763923 4993 raft_consensus.cc:1261] T 3d6859d53fd747a5a5e94722626fa522 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:35.764447 5255 raft_consensus.cc:1261] T 3d6859d53fd747a5a5e94722626fa522 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:35.765344 8383 consensus_queue.cc:1035] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:35.765870 8383 consensus_queue.cc:1035] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:35.771656 8680 tablet_copy_client.cc:523] T 2c39b53867e047faaf24cc6f87a8100c P 6dac90f01e744f9c8701a404440a4c77: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:05:35.771361 5255 raft_consensus.cc:2932] T 71bdb382b08341218c8bf2459fe65fc0 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
W20240501 14:05:35.776350 8594 log.cc:924] Time spent T 61a274b24f53458f88c2cde4f8dbe4f4 P 6dac90f01e744f9c8701a404440a4c77: Append to log took a long time: real 0.090s user 0.001s sys 0.000s
I20240501 14:05:35.779446 7024 raft_consensus.cc:1203] T e7d5f02ccba34254a4d20951878b1214 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.5->[1.6-1.6] Dedup: 1.6->[]
I20240501 14:05:35.781512 8680 tablet_bootstrap.cc:492] T 2c39b53867e047faaf24cc6f87a8100c P 6dac90f01e744f9c8701a404440a4c77: Bootstrap starting.
I20240501 14:05:35.788187 8273 raft_consensus.cc:2932] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:35.790403 4993 raft_consensus.cc:2932] T 3d6859d53fd747a5a5e94722626fa522 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:35.798272 6892 raft_consensus.cc:2932] T 3d6859d53fd747a5a5e94722626fa522 P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:35.806169 4847 catalog_manager.cc:5484] T 3d6859d53fd747a5a5e94722626fa522 P 04088ee3bea440a7bd5ff772152b9410 reported cstate change: config changed from index 4 to 5, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:35.815544 5256 raft_consensus.cc:2932] T 3d6859d53fd747a5a5e94722626fa522 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:35.823877 5125 consensus_queue.cc:237] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:05:35.827004 6892 raft_consensus.cc:1261] T 3d6859d53fd747a5a5e94722626fa522 P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:35.828682 8383 consensus_queue.cc:1035] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:35.831328 8383 raft_consensus.cc:2932] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:35.832309 5255 raft_consensus.cc:1261] T 3d6859d53fd747a5a5e94722626fa522 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:35.833014 8279 consensus_queue.cc:1035] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:35.834741 5255 raft_consensus.cc:2932] T 3d6859d53fd747a5a5e94722626fa522 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:35.835467 4839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3d6859d53fd747a5a5e94722626fa522 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:05:35.837103 6892 raft_consensus.cc:2932] T 3d6859d53fd747a5a5e94722626fa522 P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:35.839867 4854 catalog_manager.cc:5484] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 reported cstate change: config changed from index 5 to 6, VOTER 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129) evicted. New cstate: current_term: 1 leader_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:05:35.850641 4973 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d6859d53fd747a5a5e94722626fa522 with delete_type TABLET_DATA_TOMBSTONED (TS 04088ee3bea440a7bd5ff772152b9410 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41194
I20240501 14:05:35.866636 8829 tablet_replica.cc:330] T 3d6859d53fd747a5a5e94722626fa522 P 04088ee3bea440a7bd5ff772152b9410: stopping tablet replica
I20240501 14:05:35.874990 8829 raft_consensus.cc:2229] T 3d6859d53fd747a5a5e94722626fa522 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:05:35.876102 8829 raft_consensus.cc:2258] T 3d6859d53fd747a5a5e94722626fa522 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:05:35.879024 8642 tablet_bootstrap.cc:492] T 61a274b24f53458f88c2cde4f8dbe4f4 P 938a24b0e6a040b7966225af26558ff8: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:05:35.879469 8642 tablet_bootstrap.cc:492] T 61a274b24f53458f88c2cde4f8dbe4f4 P 938a24b0e6a040b7966225af26558ff8: Bootstrap complete.
I20240501 14:05:35.879743 8642 ts_tablet_manager.cc:1367] T 61a274b24f53458f88c2cde4f8dbe4f4 P 938a24b0e6a040b7966225af26558ff8: Time spent bootstrapping tablet: real 0.153s user 0.056s sys 0.010s
I20240501 14:05:35.880569 8642 raft_consensus.cc:348] T 61a274b24f53458f88c2cde4f8dbe4f4 P 938a24b0e6a040b7966225af26558ff8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }
I20240501 14:05:35.880895 8642 raft_consensus.cc:729] T 61a274b24f53458f88c2cde4f8dbe4f4 P 938a24b0e6a040b7966225af26558ff8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 938a24b0e6a040b7966225af26558ff8, State: Initialized, Role: LEARNER
I20240501 14:05:35.881151 8642 consensus_queue.cc:260] T 61a274b24f53458f88c2cde4f8dbe4f4 P 938a24b0e6a040b7966225af26558ff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }
I20240501 14:05:35.881803 8642 ts_tablet_manager.cc:1398] T 61a274b24f53458f88c2cde4f8dbe4f4 P 938a24b0e6a040b7966225af26558ff8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:05:35.883235 5406 tablet_copy_service.cc:342] P be56773105e744919601127a94e98adf: Request end of tablet copy session 938a24b0e6a040b7966225af26558ff8-61a274b24f53458f88c2cde4f8dbe4f4 received from {username='slave'} at 127.29.124.133:50583
I20240501 14:05:35.883343 8829 ts_tablet_manager.cc:1875] T 3d6859d53fd747a5a5e94722626fa522 P 04088ee3bea440a7bd5ff772152b9410: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:05:35.883383 5406 tablet_copy_service.cc:433] P be56773105e744919601127a94e98adf: ending tablet copy session 938a24b0e6a040b7966225af26558ff8-61a274b24f53458f88c2cde4f8dbe4f4 on tablet 61a274b24f53458f88c2cde4f8dbe4f4 with peer 938a24b0e6a040b7966225af26558ff8
I20240501 14:05:35.901711 8375 raft_consensus.cc:1050] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410: attempting to promote NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 to VOTER
I20240501 14:05:35.902603 8375 consensus_queue.cc:237] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } }
I20240501 14:05:35.946600 7155 raft_consensus.cc:1261] T ba6cba753e934e719c730593c0706346 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 LEARNER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:35.946782 5256 raft_consensus.cc:1261] T ba6cba753e934e719c730593c0706346 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:35.947471 8627 consensus_queue.cc:1035] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:35.947676 5125 raft_consensus.cc:1261] T ba6cba753e934e719c730593c0706346 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:35.948454 8627 consensus_queue.cc:1035] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:35.948848 8654 consensus_queue.cc:1035] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:35.953110 8277 raft_consensus.cc:1050] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c: attempting to promote NON_VOTER 938a24b0e6a040b7966225af26558ff8 to VOTER
I20240501 14:05:35.954177 8277 consensus_queue.cc:237] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:05:35.966675 8829 ts_tablet_manager.cc:1888] T 3d6859d53fd747a5a5e94722626fa522 P 04088ee3bea440a7bd5ff772152b9410: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:05:35.966929 8829 log.cc:1196] T 3d6859d53fd747a5a5e94722626fa522 P 04088ee3bea440a7bd5ff772152b9410: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/wal/wals/3d6859d53fd747a5a5e94722626fa522
I20240501 14:05:35.968000 4839 catalog_manager.cc:4830] TS 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129:44033): tablet 3d6859d53fd747a5a5e94722626fa522 (table rebalance_test_table_8 [id=f26ba2f4e5654a5fa72021df5929200d]) successfully deleted
I20240501 14:05:35.977044 8590 ts_tablet_manager.cc:902] T 43b46fcf827c4275a47291afb107c290 P 19a6d2d8a08142d5bffcde3a37234692: Initiating tablet copy from peer e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377)
I20240501 14:05:35.987939 7156 raft_consensus.cc:1261] T 51cbcf4888c440c2ad096c988ab2d968 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 LEARNER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:35.988137 4993 raft_consensus.cc:1261] T 51cbcf4888c440c2ad096c988ab2d968 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:35.988673 8659 consensus_queue.cc:1035] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:35.995091 8659 consensus_queue.cc:1035] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:36.001883 8654 raft_consensus.cc:2932] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:36.003190 6892 raft_consensus.cc:1261] T 51cbcf4888c440c2ad096c988ab2d968 P 938a24b0e6a040b7966225af26558ff8 [term 1 LEARNER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:36.003898 8659 consensus_queue.cc:1035] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:36.008337 5125 raft_consensus.cc:2932] T ba6cba753e934e719c730593c0706346 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:36.009117 7155 raft_consensus.cc:2932] T ba6cba753e934e719c730593c0706346 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:36.014123 4853 catalog_manager.cc:5484] T ba6cba753e934e719c730593c0706346 P e9c60449cf3c46f8aca4ab6c535302a4 reported cstate change: config changed from index 4 to 5, 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "04088ee3bea440a7bd5ff772152b9410" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:36.014690 8590 tablet_copy_client.cc:308] T 43b46fcf827c4275a47291afb107c290 P 19a6d2d8a08142d5bffcde3a37234692: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.130:34377
I20240501 14:05:36.018240 5382 raft_consensus.cc:1261] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:36.019009 8659 consensus_queue.cc:1035] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:36.024441 5255 consensus_queue.cc:237] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }
I20240501 14:05:36.033329 8500 raft_consensus.cc:1013] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEADER]: attempt to promote peer 4360b59e3a154688a79ca5f9c0a37bc9: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 14:05:36.035861 7286 raft_consensus.cc:1261] T de6c7753c8564107b858876fcfa5924d P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:05:36.036445 8488 consensus_queue.cc:1035] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 14:05:36.038897 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: de6c7753c8564107b858876fcfa5924d. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:36.039809 5384 raft_consensus.cc:1261] T de6c7753c8564107b858876fcfa5924d P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:05:36.041275 8352 raft_consensus.cc:1050] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf: attempting to promote NON_VOTER 938a24b0e6a040b7966225af26558ff8 to VOTER
I20240501 14:05:36.042275 8352 consensus_queue.cc:237] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:05:36.045136 8488 consensus_queue.cc:1035] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:05:36.072597 5146 tablet_copy_service.cc:140] P e9c60449cf3c46f8aca4ab6c535302a4: Received BeginTabletCopySession request for tablet 43b46fcf827c4275a47291afb107c290 from peer 19a6d2d8a08142d5bffcde3a37234692 ({username='slave'} at 127.29.124.134:41917)
I20240501 14:05:36.072820 5146 tablet_copy_service.cc:161] P e9c60449cf3c46f8aca4ab6c535302a4: Beginning new tablet copy session on tablet 43b46fcf827c4275a47291afb107c290 from peer 19a6d2d8a08142d5bffcde3a37234692 at {username='slave'} at 127.29.124.134:41917: session id = 19a6d2d8a08142d5bffcde3a37234692-43b46fcf827c4275a47291afb107c290
I20240501 14:05:36.076005 5146 tablet_copy_source_session.cc:205] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:05:36.077553 8590 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43b46fcf827c4275a47291afb107c290. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:36.084138 8590 tablet_copy_client.cc:783] T 43b46fcf827c4275a47291afb107c290 P 19a6d2d8a08142d5bffcde3a37234692: tablet copy: Starting download of 0 data blocks...
I20240501 14:05:36.084502 8590 tablet_copy_client.cc:647] T 43b46fcf827c4275a47291afb107c290 P 19a6d2d8a08142d5bffcde3a37234692: tablet copy: Starting download of 1 WAL segments...
I20240501 14:05:36.086611 8590 tablet_copy_client.cc:523] T 43b46fcf827c4275a47291afb107c290 P 19a6d2d8a08142d5bffcde3a37234692: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:05:36.091498 8488 raft_consensus.cc:1050] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c: attempting to promote NON_VOTER 938a24b0e6a040b7966225af26558ff8 to VOTER
I20240501 14:05:36.092447 8488 consensus_queue.cc:237] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } }
I20240501 14:05:36.093638 8874 ts_tablet_manager.cc:902] T 71bdb382b08341218c8bf2459fe65fc0 P 4360b59e3a154688a79ca5f9c0a37bc9: Initiating tablet copy from peer be56773105e744919601127a94e98adf (127.29.124.132:35459)
I20240501 14:05:36.095404 7287 raft_consensus.cc:1261] T 71660575d94748d8a9329f909a42a0cd P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:36.096027 6890 raft_consensus.cc:1261] T 71660575d94748d8a9329f909a42a0cd P 938a24b0e6a040b7966225af26558ff8 [term 1 LEARNER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:36.096153 8500 consensus_queue.cc:1035] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:36.096622 8659 consensus_queue.cc:1035] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:36.099279 5385 raft_consensus.cc:1261] T 71660575d94748d8a9329f909a42a0cd P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:36.100565 4992 raft_consensus.cc:1261] T 71660575d94748d8a9329f909a42a0cd P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:36.101111 8500 consensus_queue.cc:1035] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:36.103072 8590 tablet_bootstrap.cc:492] T 43b46fcf827c4275a47291afb107c290 P 19a6d2d8a08142d5bffcde3a37234692: Bootstrap starting.
I20240501 14:05:36.103205 8500 consensus_queue.cc:1035] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:36.113720 8500 raft_consensus.cc:1013] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEADER]: attempt to promote peer 6dac90f01e744f9c8701a404440a4c77: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 14:05:36.116657 8659 raft_consensus.cc:2932] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:36.122857 7287 raft_consensus.cc:2932] T 71660575d94748d8a9329f909a42a0cd P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:36.122794 5385 raft_consensus.cc:2932] T 71660575d94748d8a9329f909a42a0cd P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:36.125852 8500 raft_consensus.cc:1050] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c: attempting to promote NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 to VOTER
I20240501 14:05:36.126515 8874 tablet_copy_client.cc:308] T 71bdb382b08341218c8bf2459fe65fc0 P 4360b59e3a154688a79ca5f9c0a37bc9: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.132:35459
I20240501 14:05:36.126713 8500 consensus_queue.cc:237] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } }
I20240501 14:05:36.127941 5406 tablet_copy_service.cc:140] P be56773105e744919601127a94e98adf: Received BeginTabletCopySession request for tablet 71bdb382b08341218c8bf2459fe65fc0 from peer 4360b59e3a154688a79ca5f9c0a37bc9 ({username='slave'} at 127.29.124.135:51309)
I20240501 14:05:36.128116 5406 tablet_copy_service.cc:161] P be56773105e744919601127a94e98adf: Beginning new tablet copy session on tablet 71bdb382b08341218c8bf2459fe65fc0 from peer 4360b59e3a154688a79ca5f9c0a37bc9 at {username='slave'} at 127.29.124.135:51309: session id = 4360b59e3a154688a79ca5f9c0a37bc9-71bdb382b08341218c8bf2459fe65fc0
I20240501 14:05:36.131058 5406 tablet_copy_source_session.cc:205] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:05:36.132400 8874 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71bdb382b08341218c8bf2459fe65fc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:36.136808 4853 catalog_manager.cc:5484] T 71660575d94748d8a9329f909a42a0cd P be56773105e744919601127a94e98adf reported cstate change: config changed from index 5 to 6, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:36.142365 8612 raft_consensus.cc:1050] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf: attempting to promote NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 to VOTER
I20240501 14:05:36.143461 8612 consensus_queue.cc:237] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }
I20240501 14:05:36.143982 8676 tablet_bootstrap.cc:492] T e7d5f02ccba34254a4d20951878b1214 P 6dac90f01e744f9c8701a404440a4c77: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:05:36.144439 8676 tablet_bootstrap.cc:492] T e7d5f02ccba34254a4d20951878b1214 P 6dac90f01e744f9c8701a404440a4c77: Bootstrap complete.
I20240501 14:05:36.144754 8676 ts_tablet_manager.cc:1367] T e7d5f02ccba34254a4d20951878b1214 P 6dac90f01e744f9c8701a404440a4c77: Time spent bootstrapping tablet: real 0.544s user 0.049s sys 0.014s
I20240501 14:05:36.149796 8587 tablet_bootstrap.cc:492] T 404e2851055541f0be1bb0971a7cc72a P 19a6d2d8a08142d5bffcde3a37234692: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:05:36.150956 8587 tablet_bootstrap.cc:492] T 404e2851055541f0be1bb0971a7cc72a P 19a6d2d8a08142d5bffcde3a37234692: Bootstrap complete.
I20240501 14:05:36.155066 8594 tablet_bootstrap.cc:492] T 61a274b24f53458f88c2cde4f8dbe4f4 P 6dac90f01e744f9c8701a404440a4c77: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:05:36.156929 6891 raft_consensus.cc:1261] T 61a274b24f53458f88c2cde4f8dbe4f4 P 938a24b0e6a040b7966225af26558ff8 [term 1 LEARNER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:05:36.157521 8503 consensus_queue.cc:1035] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 14:05:36.160231 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:36.161289 7024 raft_consensus.cc:1261] T 61a274b24f53458f88c2cde4f8dbe4f4 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 LEARNER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:05:36.161589 5255 raft_consensus.cc:1261] T 61a274b24f53458f88c2cde4f8dbe4f4 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:05:36.161955 5125 raft_consensus.cc:1261] T 61a274b24f53458f88c2cde4f8dbe4f4 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:05:36.162242 8503 consensus_queue.cc:1035] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:05:36.162569 8804 consensus_queue.cc:1035] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:05:36.163152 8503 consensus_queue.cc:1035] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:05:36.167600 8503 raft_consensus.cc:2932] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:36.173563 8874 tablet_copy_client.cc:783] T 71bdb382b08341218c8bf2459fe65fc0 P 4360b59e3a154688a79ca5f9c0a37bc9: tablet copy: Starting download of 0 data blocks...
I20240501 14:05:36.172912 7024 raft_consensus.cc:2932] T 61a274b24f53458f88c2cde4f8dbe4f4 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:36.174090 8874 tablet_copy_client.cc:647] T 71bdb382b08341218c8bf2459fe65fc0 P 4360b59e3a154688a79ca5f9c0a37bc9: tablet copy: Starting download of 1 WAL segments...
I20240501 14:05:36.176249 8874 tablet_copy_client.cc:523] T 71bdb382b08341218c8bf2459fe65fc0 P 4360b59e3a154688a79ca5f9c0a37bc9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:05:36.176249 8587 ts_tablet_manager.cc:1367] T 404e2851055541f0be1bb0971a7cc72a P 19a6d2d8a08142d5bffcde3a37234692: Time spent bootstrapping tablet: real 0.577s user 0.056s sys 0.016s
I20240501 14:05:36.177146 8587 raft_consensus.cc:348] T 404e2851055541f0be1bb0971a7cc72a P 19a6d2d8a08142d5bffcde3a37234692 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } }
I20240501 14:05:36.180594 8587 raft_consensus.cc:729] T 404e2851055541f0be1bb0971a7cc72a P 19a6d2d8a08142d5bffcde3a37234692 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 19a6d2d8a08142d5bffcde3a37234692, State: Initialized, Role: LEARNER
I20240501 14:05:36.182992 4853 catalog_manager.cc:5484] T 61a274b24f53458f88c2cde4f8dbe4f4 P 19a6d2d8a08142d5bffcde3a37234692 reported cstate change: config changed from index 6 to 7, 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:36.183851 8874 tablet_bootstrap.cc:492] T 71bdb382b08341218c8bf2459fe65fc0 P 4360b59e3a154688a79ca5f9c0a37bc9: Bootstrap starting.
I20240501 14:05:36.183979 8587 consensus_queue.cc:260] T 404e2851055541f0be1bb0971a7cc72a P 19a6d2d8a08142d5bffcde3a37234692 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } }
I20240501 14:05:36.186462 8804 raft_consensus.cc:1050] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf: attempting to promote NON_VOTER 938a24b0e6a040b7966225af26558ff8 to VOTER
I20240501 14:05:36.186348 6891 raft_consensus.cc:2932] T 61a274b24f53458f88c2cde4f8dbe4f4 P 938a24b0e6a040b7966225af26558ff8 [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:36.187348 8804 consensus_queue.cc:237] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:05:36.187191 5125 raft_consensus.cc:2932] T 61a274b24f53458f88c2cde4f8dbe4f4 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
W20240501 14:05:36.190834 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:36.194507 6889 raft_consensus.cc:1261] T 61a274b24f53458f88c2cde4f8dbe4f4 P 938a24b0e6a040b7966225af26558ff8 [term 1 LEARNER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:05:36.195760 8587 ts_tablet_manager.cc:1398] T 404e2851055541f0be1bb0971a7cc72a P 19a6d2d8a08142d5bffcde3a37234692: Time spent starting tablet: real 0.019s user 0.002s sys 0.001s
I20240501 14:05:36.197046 5126 raft_consensus.cc:1261] T 61a274b24f53458f88c2cde4f8dbe4f4 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:05:36.197331 8594 tablet_bootstrap.cc:492] T 61a274b24f53458f88c2cde4f8dbe4f4 P 6dac90f01e744f9c8701a404440a4c77: Bootstrap complete.
I20240501 14:05:36.197325 7024 raft_consensus.cc:1261] T 61a274b24f53458f88c2cde4f8dbe4f4 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:05:36.197626 8594 ts_tablet_manager.cc:1367] T 61a274b24f53458f88c2cde4f8dbe4f4 P 6dac90f01e744f9c8701a404440a4c77: Time spent bootstrapping tablet: real 0.551s user 0.052s sys 0.000s
I20240501 14:05:36.197695 5276 tablet_copy_service.cc:342] P 1c3fdc5272484ac09ef3e5e36a08f27c: Request end of tablet copy session 19a6d2d8a08142d5bffcde3a37234692-404e2851055541f0be1bb0971a7cc72a received from {username='slave'} at 127.29.124.134:43435
I20240501 14:05:36.197818 5276 tablet_copy_service.cc:433] P 1c3fdc5272484ac09ef3e5e36a08f27c: ending tablet copy session 19a6d2d8a08142d5bffcde3a37234692-404e2851055541f0be1bb0971a7cc72a on tablet 404e2851055541f0be1bb0971a7cc72a with peer 19a6d2d8a08142d5bffcde3a37234692
I20240501 14:05:36.198518 8594 raft_consensus.cc:348] T 61a274b24f53458f88c2cde4f8dbe4f4 P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }
I20240501 14:05:36.198812 8594 raft_consensus.cc:729] T 61a274b24f53458f88c2cde4f8dbe4f4 P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6dac90f01e744f9c8701a404440a4c77, State: Initialized, Role: LEARNER
I20240501 14:05:36.199041 8594 consensus_queue.cc:260] T 61a274b24f53458f88c2cde4f8dbe4f4 P 6dac90f01e744f9c8701a404440a4c77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }
I20240501 14:05:36.156401 8676 raft_consensus.cc:348] T e7d5f02ccba34254a4d20951878b1214 P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }
I20240501 14:05:36.199456 8676 raft_consensus.cc:729] T e7d5f02ccba34254a4d20951878b1214 P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6dac90f01e744f9c8701a404440a4c77, State: Initialized, Role: LEARNER
I20240501 14:05:36.199661 8676 consensus_queue.cc:260] T e7d5f02ccba34254a4d20951878b1214 P 6dac90f01e744f9c8701a404440a4c77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }
I20240501 14:05:36.200389 8804 consensus_queue.cc:1035] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:05:36.201617 8804 consensus_queue.cc:1035] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:05:36.202697 8804 consensus_queue.cc:1035] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:05:36.204460 8247 raft_consensus.cc:2932] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:36.205826 5126 raft_consensus.cc:2932] T 61a274b24f53458f88c2cde4f8dbe4f4 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:36.208808 8676 ts_tablet_manager.cc:1398] T e7d5f02ccba34254a4d20951878b1214 P 6dac90f01e744f9c8701a404440a4c77: Time spent starting tablet: real 0.053s user 0.002s sys 0.000s
I20240501 14:05:36.209460 8594 ts_tablet_manager.cc:1398] T 61a274b24f53458f88c2cde4f8dbe4f4 P 6dac90f01e744f9c8701a404440a4c77: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240501 14:05:36.209599 5406 tablet_copy_service.cc:342] P be56773105e744919601127a94e98adf: Request end of tablet copy session 6dac90f01e744f9c8701a404440a4c77-e7d5f02ccba34254a4d20951878b1214 received from {username='slave'} at 127.29.124.136:50719
I20240501 14:05:36.209712 5406 tablet_copy_service.cc:433] P be56773105e744919601127a94e98adf: ending tablet copy session 6dac90f01e744f9c8701a404440a4c77-e7d5f02ccba34254a4d20951878b1214 on tablet e7d5f02ccba34254a4d20951878b1214 with peer 6dac90f01e744f9c8701a404440a4c77
I20240501 14:05:36.209545 6891 raft_consensus.cc:2932] T 61a274b24f53458f88c2cde4f8dbe4f4 P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:36.210813 5406 tablet_copy_service.cc:342] P be56773105e744919601127a94e98adf: Request end of tablet copy session 6dac90f01e744f9c8701a404440a4c77-61a274b24f53458f88c2cde4f8dbe4f4 received from {username='slave'} at 127.29.124.136:50719
I20240501 14:05:36.210934 5406 tablet_copy_service.cc:433] P be56773105e744919601127a94e98adf: ending tablet copy session 6dac90f01e744f9c8701a404440a4c77-61a274b24f53458f88c2cde4f8dbe4f4 on tablet 61a274b24f53458f88c2cde4f8dbe4f4 with peer 6dac90f01e744f9c8701a404440a4c77
I20240501 14:05:36.213274 4853 catalog_manager.cc:5484] T 61a274b24f53458f88c2cde4f8dbe4f4 P e9c60449cf3c46f8aca4ab6c535302a4 reported cstate change: config changed from index 7 to 8, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:36.219136 7024 raft_consensus.cc:2932] T 61a274b24f53458f88c2cde4f8dbe4f4 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:36.221686 5385 consensus_queue.cc:237] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:05:36.223397 8680 tablet_bootstrap.cc:492] T 2c39b53867e047faaf24cc6f87a8100c P 6dac90f01e744f9c8701a404440a4c77: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:05:36.223824 8680 tablet_bootstrap.cc:492] T 2c39b53867e047faaf24cc6f87a8100c P 6dac90f01e744f9c8701a404440a4c77: Bootstrap complete.
I20240501 14:05:36.224864 7287 raft_consensus.cc:1261] T 61a274b24f53458f88c2cde4f8dbe4f4 P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:05:36.225503 8247 consensus_queue.cc:1035] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:05:36.225823 7025 raft_consensus.cc:1261] T 61a274b24f53458f88c2cde4f8dbe4f4 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:05:36.226387 8503 consensus_queue.cc:1035] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:05:36.226604 5126 raft_consensus.cc:1261] T 61a274b24f53458f88c2cde4f8dbe4f4 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:05:36.227870 7287 raft_consensus.cc:858] T 61a274b24f53458f88c2cde4f8dbe4f4 P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }.
Pending config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }:
New config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
W20240501 14:05:36.228006 7287 raft_consensus.cc:1565] T 61a274b24f53458f88c2cde4f8dbe4f4 P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Could not prepare op 'term: 1 index: 8' and following 1 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }.
Pending config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }:
New config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:05:36.228199 8612 consensus_queue.cc:1035] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:05:36.232758 8247 raft_consensus.cc:2932] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:36.226112 6891 raft_consensus.cc:1261] T 61a274b24f53458f88c2cde4f8dbe4f4 P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:05:36.234472 8804 consensus_queue.cc:1035] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:05:36.236074 6889 raft_consensus.cc:2932] T 61a274b24f53458f88c2cde4f8dbe4f4 P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:36.240952 4839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 61a274b24f53458f88c2cde4f8dbe4f4 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:05:36.242344 5126 raft_consensus.cc:2932] T 61a274b24f53458f88c2cde4f8dbe4f4 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:36.244364 4852 catalog_manager.cc:5484] T 61a274b24f53458f88c2cde4f8dbe4f4 P 938a24b0e6a040b7966225af26558ff8 reported cstate change: config changed from index 8 to 9, VOTER 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) evicted. New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:36.248275 7024 raft_consensus.cc:2932] T 61a274b24f53458f88c2cde4f8dbe4f4 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:36.250046 8680 ts_tablet_manager.cc:1367] T 2c39b53867e047faaf24cc6f87a8100c P 6dac90f01e744f9c8701a404440a4c77: Time spent bootstrapping tablet: real 0.474s user 0.050s sys 0.010s
I20240501 14:05:36.254658 5236 tablet_service.cc:1514] Processing DeleteTablet for tablet 61a274b24f53458f88c2cde4f8dbe4f4 with delete_type TABLET_DATA_TOMBSTONED (TS 1c3fdc5272484ac09ef3e5e36a08f27c not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:39020
I20240501 14:05:36.256325 8680 raft_consensus.cc:348] T 2c39b53867e047faaf24cc6f87a8100c P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } }
I20240501 14:05:36.256599 8680 raft_consensus.cc:729] T 2c39b53867e047faaf24cc6f87a8100c P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6dac90f01e744f9c8701a404440a4c77, State: Initialized, Role: LEARNER
I20240501 14:05:36.256831 8680 consensus_queue.cc:260] T 2c39b53867e047faaf24cc6f87a8100c P 6dac90f01e744f9c8701a404440a4c77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } }
I20240501 14:05:36.261968 8680 ts_tablet_manager.cc:1398] T 2c39b53867e047faaf24cc6f87a8100c P 6dac90f01e744f9c8701a404440a4c77: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240501 14:05:36.261838 8488 raft_consensus.cc:2932] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:36.262660 5406 tablet_copy_service.cc:342] P be56773105e744919601127a94e98adf: Request end of tablet copy session 6dac90f01e744f9c8701a404440a4c77-2c39b53867e047faaf24cc6f87a8100c received from {username='slave'} at 127.29.124.136:50719
I20240501 14:05:36.262776 5406 tablet_copy_service.cc:433] P be56773105e744919601127a94e98adf: ending tablet copy session 6dac90f01e744f9c8701a404440a4c77-2c39b53867e047faaf24cc6f87a8100c on tablet 2c39b53867e047faaf24cc6f87a8100c with peer 6dac90f01e744f9c8701a404440a4c77
I20240501 14:05:36.267880 7156 raft_consensus.cc:2932] T 51cbcf4888c440c2ad096c988ab2d968 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:36.272290 8277 raft_consensus.cc:2932] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:36.274631 4852 catalog_manager.cc:5484] T 51cbcf4888c440c2ad096c988ab2d968 P 4360b59e3a154688a79ca5f9c0a37bc9 reported cstate change: config changed from index 5 to 6, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:36.277750 7285 raft_consensus.cc:1261] T e7d5f02ccba34254a4d20951878b1214 P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:05:36.279043 7024 raft_consensus.cc:1261] T e7d5f02ccba34254a4d20951878b1214 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 LEARNER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:05:36.279671 8503 consensus_queue.cc:1035] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.002s
I20240501 14:05:36.282058 8891 consensus_queue.cc:1035] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:05:36.282744 6892 raft_consensus.cc:1261] T e7d5f02ccba34254a4d20951878b1214 P 938a24b0e6a040b7966225af26558ff8 [term 1 LEARNER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:05:36.283442 8891 consensus_queue.cc:1035] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:05:36.283459 4993 raft_consensus.cc:2932] T 51cbcf4888c440c2ad096c988ab2d968 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:36.295501 5384 raft_consensus.cc:2932] T de6c7753c8564107b858876fcfa5924d P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:36.306807 4992 raft_consensus.cc:1261] T e7d5f02ccba34254a4d20951878b1214 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:05:36.307600 8804 consensus_queue.cc:1035] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:05:36.311256 5382 raft_consensus.cc:2932] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:36.332126 7156 raft_consensus.cc:1261] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 LEARNER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:36.332728 8804 raft_consensus.cc:1013] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [term 1 LEADER]: attempt to promote peer 19a6d2d8a08142d5bffcde3a37234692: there is already a config change operation in progress. Unable to promote follower until it completes. Doing nothing.
I20240501 14:05:36.333974 6890 raft_consensus.cc:2932] T 71660575d94748d8a9329f909a42a0cd P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:36.334864 8905 consensus_queue.cc:1035] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:36.339056 4992 raft_consensus.cc:2932] T 71660575d94748d8a9329f909a42a0cd P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:36.344740 5126 raft_consensus.cc:1261] T ec1e81011c6a46bc8a39aecf3a5e71e4 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:36.344960 4995 raft_consensus.cc:1261] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:36.345645 8904 consensus_queue.cc:1035] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:36.354055 8503 raft_consensus.cc:2932] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:36.356063 8893 tablet_replica.cc:330] T 61a274b24f53458f88c2cde4f8dbe4f4 P 1c3fdc5272484ac09ef3e5e36a08f27c: stopping tablet replica
I20240501 14:05:36.356261 8893 raft_consensus.cc:2229] T 61a274b24f53458f88c2cde4f8dbe4f4 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:05:36.356580 8893 pending_rounds.cc:62] T 61a274b24f53458f88c2cde4f8dbe4f4 P 1c3fdc5272484ac09ef3e5e36a08f27c: Trying to abort 1 pending ops.
I20240501 14:05:36.356684 8893 pending_rounds.cc:68] T 61a274b24f53458f88c2cde4f8dbe4f4 P 1c3fdc5272484ac09ef3e5e36a08f27c: Aborting op as it isn't in flight: id { term: 1 index: 7 } timestamp: 7022888288840732672 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "61a274b24f53458f88c2cde4f8dbe4f4" old_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } } new_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } } }
I20240501 14:05:36.357295 8893 raft_consensus.cc:2894] T 61a274b24f53458f88c2cde4f8dbe4f4 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Aborting config change with OpId 1.7: Aborted: Op aborted
I20240501 14:05:36.357410 8893 raft_consensus.cc:2866] T 61a274b24f53458f88c2cde4f8dbe4f4 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 14:05:36.357515 8893 raft_consensus.cc:2258] T 61a274b24f53458f88c2cde4f8dbe4f4 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:05:36.338752 8659 raft_consensus.cc:1050] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c: attempting to promote NON_VOTER 6dac90f01e744f9c8701a404440a4c77 to VOTER
I20240501 14:05:36.359778 4991 raft_consensus.cc:2932] T e7d5f02ccba34254a4d20951878b1214 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:36.361523 8500 consensus_queue.cc:1035] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:36.365190 7024 raft_consensus.cc:2932] T e7d5f02ccba34254a4d20951878b1214 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:36.370803 8488 raft_consensus.cc:1050] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c: attempting to promote NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 to VOTER
I20240501 14:05:36.371151 6891 raft_consensus.cc:2932] T 51cbcf4888c440c2ad096c988ab2d968 P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:36.371608 8488 consensus_queue.cc:237] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:05:36.374476 8804 raft_consensus.cc:1050] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf: attempting to promote NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 to VOTER
I20240501 14:05:36.375312 8804 consensus_queue.cc:237] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:05:36.375831 4846 catalog_manager.cc:5484] T e7d5f02ccba34254a4d20951878b1214 P 19a6d2d8a08142d5bffcde3a37234692 reported cstate change: config changed from index 6 to 7, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:36.358536 5255 raft_consensus.cc:2932] T ba6cba753e934e719c730593c0706346 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:36.378945 7025 raft_consensus.cc:1261] T e7d5f02ccba34254a4d20951878b1214 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 LEARNER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:05:36.380084 8247 consensus_queue.cc:1035] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:05:36.380226 7156 raft_consensus.cc:1261] T 51cbcf4888c440c2ad096c988ab2d968 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 LEARNER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:05:36.381477 8895 consensus_queue.cc:1035] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:05:36.382685 4994 raft_consensus.cc:1261] T e7d5f02ccba34254a4d20951878b1214 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:05:36.383275 8804 consensus_queue.cc:1035] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:05:36.361033 8659 consensus_queue.cc:237] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } }
I20240501 14:05:36.386484 6888 raft_consensus.cc:1261] T 51cbcf4888c440c2ad096c988ab2d968 P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:05:36.387796 8503 raft_consensus.cc:2932] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:36.389768 8893 ts_tablet_manager.cc:1875] T 61a274b24f53458f88c2cde4f8dbe4f4 P 1c3fdc5272484ac09ef3e5e36a08f27c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:05:36.392481 8500 consensus_queue.cc:1035] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:05:36.392398 7025 raft_consensus.cc:2932] T e7d5f02ccba34254a4d20951878b1214 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:36.373430 6892 raft_consensus.cc:2932] T e7d5f02ccba34254a4d20951878b1214 P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:36.397456 8893 ts_tablet_manager.cc:1888] T 61a274b24f53458f88c2cde4f8dbe4f4 P 1c3fdc5272484ac09ef3e5e36a08f27c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240501 14:05:36.397656 8893 log.cc:1196] T 61a274b24f53458f88c2cde4f8dbe4f4 P 1c3fdc5272484ac09ef3e5e36a08f27c: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/wal/wals/61a274b24f53458f88c2cde4f8dbe4f4
I20240501 14:05:36.398495 4838 catalog_manager.cc:4830] TS 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209): tablet 61a274b24f53458f88c2cde4f8dbe4f4 (table rebalance_test_table_9 [id=5fc33bdfba464f65b68de5887d425ab7]) successfully deleted
I20240501 14:05:36.401783 4846 catalog_manager.cc:5484] T e7d5f02ccba34254a4d20951878b1214 P 19a6d2d8a08142d5bffcde3a37234692 reported cstate change: config changed from index 7 to 8, 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:36.401329 8905 raft_consensus.cc:2932] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:36.403823 4994 raft_consensus.cc:2932] T e7d5f02ccba34254a4d20951878b1214 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:36.405620 5126 raft_consensus.cc:2932] T ec1e81011c6a46bc8a39aecf3a5e71e4 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:36.411408 4846 catalog_manager.cc:5484] T ec1e81011c6a46bc8a39aecf3a5e71e4 P e9c60449cf3c46f8aca4ab6c535302a4 reported cstate change: config changed from index 4 to 5, 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:36.411933 4852 catalog_manager.cc:5484] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c reported cstate change: config changed from index 6 to 7, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New cstate: current_term: 1 leader_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:05:36.412783 7286 raft_consensus.cc:1261] T e7d5f02ccba34254a4d20951878b1214 P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:05:36.413580 8804 consensus_queue.cc:1035] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:05:36.415071 7284 raft_consensus.cc:2932] T e7d5f02ccba34254a4d20951878b1214 P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:36.417701 4991 raft_consensus.cc:1261] T 51cbcf4888c440c2ad096c988ab2d968 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:05:36.418426 8904 consensus_queue.cc:1035] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:05:36.420322 7284 raft_consensus.cc:2932] T e7d5f02ccba34254a4d20951878b1214 P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:36.422505 4994 raft_consensus.cc:2932] T 51cbcf4888c440c2ad096c988ab2d968 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
W20240501 14:05:36.424670 5192 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.124.136:43377, user_credentials={real_user=slave}} blocked reactor thread for 146340us
I20240501 14:05:36.425024 7287 raft_consensus.cc:2932] T 61a274b24f53458f88c2cde4f8dbe4f4 P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:36.425160 8500 raft_consensus.cc:2932] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:36.432474 6888 raft_consensus.cc:2932] T 51cbcf4888c440c2ad096c988ab2d968 P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:36.434607 5251 raft_consensus.cc:1261] T e7d5f02ccba34254a4d20951878b1214 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:05:36.436059 8352 raft_consensus.cc:1050] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf: attempting to promote NON_VOTER 6dac90f01e744f9c8701a404440a4c77 to VOTER
I20240501 14:05:36.437801 8352 consensus_queue.cc:237] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 4, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:05:36.439421 4852 catalog_manager.cc:5484] T 51cbcf4888c440c2ad096c988ab2d968 P 938a24b0e6a040b7966225af26558ff8 reported cstate change: config changed from index 6 to 7, 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:36.440968 5384 raft_consensus.cc:1261] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:05:36.441589 8897 consensus_queue.cc:1035] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:05:36.442395 7284 raft_consensus.cc:1261] T e7d5f02ccba34254a4d20951878b1214 P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:05:36.442962 8804 consensus_queue.cc:1035] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:05:36.444679 7024 raft_consensus.cc:1261] T e7d5f02ccba34254a4d20951878b1214 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:05:36.445828 8503 consensus_queue.cc:1035] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:05:36.446659 5382 raft_consensus.cc:1261] T 71660575d94748d8a9329f909a42a0cd P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:05:36.447234 4994 raft_consensus.cc:1261] T 71660575d94748d8a9329f909a42a0cd P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:05:36.447595 8909 consensus_queue.cc:1035] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:05:36.448606 8911 consensus_queue.cc:1035] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:05:36.449671 5251 raft_consensus.cc:1261] T e7d5f02ccba34254a4d20951878b1214 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:05:36.452358 5385 raft_consensus.cc:2932] T 71660575d94748d8a9329f909a42a0cd P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:36.453040 6887 raft_consensus.cc:1261] T 71660575d94748d8a9329f909a42a0cd P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:05:36.453119 8911 raft_consensus.cc:2932] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:36.453779 5251 raft_consensus.cc:1261] T e7d5f02ccba34254a4d20951878b1214 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:05:36.454226 8659 consensus_queue.cc:1035] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.001s
I20240501 14:05:36.454557 7284 raft_consensus.cc:2932] T de6c7753c8564107b858876fcfa5924d P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:36.455668 6887 raft_consensus.cc:2932] T 71660575d94748d8a9329f909a42a0cd P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:36.460986 4853 catalog_manager.cc:5484] T 71660575d94748d8a9329f909a42a0cd P 938a24b0e6a040b7966225af26558ff8 reported cstate change: config changed from index 6 to 7, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:36.461710 7285 raft_consensus.cc:1261] T 71660575d94748d8a9329f909a42a0cd P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:05:36.461534 5384 raft_consensus.cc:2932] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:36.463269 4993 raft_consensus.cc:1261] T e7d5f02ccba34254a4d20951878b1214 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:05:36.463815 8503 consensus_queue.cc:1035] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:05:36.466351 8804 raft_consensus.cc:2932] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:36.466924 8352 consensus_queue.cc:1035] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:05:36.470923 5251 raft_consensus.cc:858] T e7d5f02ccba34254a4d20951878b1214 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }.
Pending config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }:
New config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
W20240501 14:05:36.471053 5251 raft_consensus.cc:1565] T e7d5f02ccba34254a4d20951878b1214 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Could not prepare op 'term: 1 index: 8' and following 1 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }.
Pending config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }:
New config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:05:36.471911 4992 raft_consensus.cc:2932] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:36.477959 7024 raft_consensus.cc:2932] T e7d5f02ccba34254a4d20951878b1214 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:36.480645 5251 raft_consensus.cc:2932] T e7d5f02ccba34254a4d20951878b1214 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:36.482196 7156 raft_consensus.cc:2932] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:36.486052 4993 raft_consensus.cc:2932] T e7d5f02ccba34254a4d20951878b1214 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:36.486853 4994 raft_consensus.cc:2932] T 71660575d94748d8a9329f909a42a0cd P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:36.491503 6889 raft_consensus.cc:1261] T e7d5f02ccba34254a4d20951878b1214 P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:05:36.492694 6888 raft_consensus.cc:1261] T e7d5f02ccba34254a4d20951878b1214 P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:05:36.493409 8352 consensus_queue.cc:1035] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:05:36.496146 8904 consensus_queue.cc:1035] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:05:36.496399 8590 tablet_bootstrap.cc:492] T 43b46fcf827c4275a47291afb107c290 P 19a6d2d8a08142d5bffcde3a37234692: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:05:36.496819 8590 tablet_bootstrap.cc:492] T 43b46fcf827c4275a47291afb107c290 P 19a6d2d8a08142d5bffcde3a37234692: Bootstrap complete.
I20240501 14:05:36.497095 8590 ts_tablet_manager.cc:1367] T 43b46fcf827c4275a47291afb107c290 P 19a6d2d8a08142d5bffcde3a37234692: Time spent bootstrapping tablet: real 0.394s user 0.055s sys 0.052s
I20240501 14:05:36.497174 5251 consensus_queue.cc:237] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } }
I20240501 14:05:36.497936 8590 raft_consensus.cc:348] T 43b46fcf827c4275a47291afb107c290 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } }
I20240501 14:05:36.498198 8590 raft_consensus.cc:729] T 43b46fcf827c4275a47291afb107c290 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 19a6d2d8a08142d5bffcde3a37234692, State: Initialized, Role: LEARNER
I20240501 14:05:36.499236 4994 consensus_queue.cc:237] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } }
I20240501 14:05:36.499677 7156 raft_consensus.cc:2932] T 51cbcf4888c440c2ad096c988ab2d968 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:36.479228 7285 raft_consensus.cc:2932] T e7d5f02ccba34254a4d20951878b1214 P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:36.503926 7155 raft_consensus.cc:1261] T ba6cba753e934e719c730593c0706346 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:36.504474 8375 consensus_queue.cc:1035] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:36.504175 4853 catalog_manager.cc:5484] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf reported cstate change: config changed from index 8 to 9, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:05:36.507007 8375 raft_consensus.cc:2932] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:36.509908 7155 raft_consensus.cc:2932] T ba6cba753e934e719c730593c0706346 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:36.511000 6892 raft_consensus.cc:858] T e7d5f02ccba34254a4d20951878b1214 P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }.
Pending config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }:
New config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
W20240501 14:05:36.511130 6892 raft_consensus.cc:1565] T e7d5f02ccba34254a4d20951878b1214 P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Could not prepare op 'term: 1 index: 9' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }.
Pending config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }:
New config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:05:36.511848 6892 raft_consensus.cc:2932] T e7d5f02ccba34254a4d20951878b1214 P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:36.515120 4839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ba6cba753e934e719c730593c0706346 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:05:36.517313 5382 consensus_queue.cc:237] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:05:36.517107 4854 catalog_manager.cc:5484] T ba6cba753e934e719c730593c0706346 P 4360b59e3a154688a79ca5f9c0a37bc9 reported cstate change: config changed from index 5 to 6, VOTER e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130) evicted. New cstate: current_term: 1 leader_uuid: "04088ee3bea440a7bd5ff772152b9410" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:36.522125 5106 tablet_service.cc:1514] Processing DeleteTablet for tablet ba6cba753e934e719c730593c0706346 with delete_type TABLET_DATA_TOMBSTONED (TS e9c60449cf3c46f8aca4ab6c535302a4 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41850
I20240501 14:05:36.523051 5255 consensus_queue.cc:237] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } }
I20240501 14:05:36.527549 8925 ts_tablet_manager.cc:902] T de6c7753c8564107b858876fcfa5924d P 938a24b0e6a040b7966225af26558ff8: Initiating tablet copy from peer 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209)
I20240501 14:05:36.528376 5251 raft_consensus.cc:1261] T ba6cba753e934e719c730593c0706346 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:36.528930 8375 consensus_queue.cc:1035] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:36.529891 7155 raft_consensus.cc:1261] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:36.530738 8895 consensus_queue.cc:1035] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:36.531373 6889 raft_consensus.cc:1261] T 71660575d94748d8a9329f909a42a0cd P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:05:36.532081 8905 consensus_queue.cc:1035] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:05:36.534112 8909 raft_consensus.cc:2932] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:36.537482 4838 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ec1e81011c6a46bc8a39aecf3a5e71e4 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:05:36.538723 7155 raft_consensus.cc:2932] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:36.545478 4994 raft_consensus.cc:1261] T 71660575d94748d8a9329f909a42a0cd P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:05:36.546047 8882 consensus_queue.cc:1035] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:05:36.546499 4851 catalog_manager.cc:5484] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 4360b59e3a154688a79ca5f9c0a37bc9 reported cstate change: config changed from index 5 to 6, VOTER e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130) evicted. New cstate: current_term: 1 leader_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:36.537181 5251 raft_consensus.cc:2932] T ba6cba753e934e719c730593c0706346 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:36.549239 4995 raft_consensus.cc:2932] T 71660575d94748d8a9329f909a42a0cd P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER be56773105e744919601127a94e98adf (127.29.124.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:36.549592 8895 raft_consensus.cc:2932] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER be56773105e744919601127a94e98adf (127.29.124.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:36.555171 4838 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 71660575d94748d8a9329f909a42a0cd with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:05:36.556123 4854 catalog_manager.cc:5484] T 71660575d94748d8a9329f909a42a0cd P 04088ee3bea440a7bd5ff772152b9410 reported cstate change: config changed from index 7 to 8, VOTER be56773105e744919601127a94e98adf (127.29.124.132) evicted. New cstate: current_term: 1 leader_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:36.559813 5254 raft_consensus.cc:1261] T e7d5f02ccba34254a4d20951878b1214 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:05:36.560956 7286 raft_consensus.cc:2932] T 71660575d94748d8a9329f909a42a0cd P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:36.563232 5256 raft_consensus.cc:858] T e7d5f02ccba34254a4d20951878b1214 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }.
Pending config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }:
New config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
W20240501 14:05:36.563362 5256 raft_consensus.cc:1565] T e7d5f02ccba34254a4d20951878b1214 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Could not prepare op 'term: 1 index: 9' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }.
Pending config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }:
New config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:05:36.563746 5256 raft_consensus.cc:2932] T e7d5f02ccba34254a4d20951878b1214 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:36.566332 5365 tablet_service.cc:1514] Processing DeleteTablet for tablet 71660575d94748d8a9329f909a42a0cd with delete_type TABLET_DATA_TOMBSTONED (TS be56773105e744919601127a94e98adf not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:43020
I20240501 14:05:36.566955 8936 tablet_replica.cc:330] T 71660575d94748d8a9329f909a42a0cd P be56773105e744919601127a94e98adf: stopping tablet replica
I20240501 14:05:36.567215 8936 raft_consensus.cc:2229] T 71660575d94748d8a9329f909a42a0cd P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:05:36.567974 8936 raft_consensus.cc:2258] T 71660575d94748d8a9329f909a42a0cd P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:05:36.570281 8936 ts_tablet_manager.cc:1875] T 71660575d94748d8a9329f909a42a0cd P be56773105e744919601127a94e98adf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:05:36.558915 6890 raft_consensus.cc:2932] T 71660575d94748d8a9329f909a42a0cd P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER be56773105e744919601127a94e98adf (127.29.124.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:36.580847 7284 raft_consensus.cc:1261] T e7d5f02ccba34254a4d20951878b1214 P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:05:36.581542 8352 consensus_queue.cc:1035] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:05:36.586438 5105 tablet_service.cc:1514] Processing DeleteTablet for tablet ec1e81011c6a46bc8a39aecf3a5e71e4 with delete_type TABLET_DATA_TOMBSTONED (TS e9c60449cf3c46f8aca4ab6c535302a4 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41850
I20240501 14:05:36.588232 7286 raft_consensus.cc:858] T 61a274b24f53458f88c2cde4f8dbe4f4 P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }.
Pending config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }:
New config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:05:36.588189 5256 consensus_queue.cc:237] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
W20240501 14:05:36.588364 7286 raft_consensus.cc:1565] T 61a274b24f53458f88c2cde4f8dbe4f4 P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Could not prepare op 'term: 1 index: 9' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }.
Pending config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }:
New config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:05:36.589092 7286 raft_consensus.cc:2932] T 61a274b24f53458f88c2cde4f8dbe4f4 P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:36.591220 7155 raft_consensus.cc:1261] T 51cbcf4888c440c2ad096c988ab2d968 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:05:36.591842 6889 raft_consensus.cc:1261] T 51cbcf4888c440c2ad096c988ab2d968 P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:05:36.592844 8905 consensus_queue.cc:1035] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:05:36.593236 8352 consensus_queue.cc:1035] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:05:36.593097 8897 consensus_queue.cc:1035] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:05:36.594604 7023 raft_consensus.cc:1203] T 404e2851055541f0be1bb0971a7cc72a P 19a6d2d8a08142d5bffcde3a37234692 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:05:36.595229 7284 raft_consensus.cc:1261] T 71660575d94748d8a9329f909a42a0cd P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:05:36.597105 8905 raft_consensus.cc:2932] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:36.599309 8904 consensus_queue.cc:1035] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:05:36.600209 5256 raft_consensus.cc:1203] T e7d5f02ccba34254a4d20951878b1214 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.7->[1.8-1.10] Dedup: 1.8->[1.9-1.10]
I20240501 14:05:36.601150 5256 raft_consensus.cc:858] T e7d5f02ccba34254a4d20951878b1214 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }.
Pending config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }:
New config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
W20240501 14:05:36.601364 5256 raft_consensus.cc:1565] T e7d5f02ccba34254a4d20951878b1214 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Could not prepare op 'term: 1 index: 10' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }.
Pending config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }:
New config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:05:36.601868 5256 raft_consensus.cc:2932] T e7d5f02ccba34254a4d20951878b1214 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:36.602429 7155 raft_consensus.cc:2932] T 51cbcf4888c440c2ad096c988ab2d968 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:36.605790 8936 ts_tablet_manager.cc:1888] T 71660575d94748d8a9329f909a42a0cd P be56773105e744919601127a94e98adf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240501 14:05:36.605650 6889 raft_consensus.cc:2932] T 51cbcf4888c440c2ad096c988ab2d968 P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:36.612423 7287 raft_consensus.cc:2932] T 71660575d94748d8a9329f909a42a0cd P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER be56773105e744919601127a94e98adf (127.29.124.132) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:36.613109 4838 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 51cbcf4888c440c2ad096c988ab2d968 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:05:36.615509 8936 log.cc:1196] T 71660575d94748d8a9329f909a42a0cd P be56773105e744919601127a94e98adf: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/wal/wals/71660575d94748d8a9329f909a42a0cd
I20240501 14:05:36.617115 4839 catalog_manager.cc:4830] TS be56773105e744919601127a94e98adf (127.29.124.132:35459): tablet 71660575d94748d8a9329f909a42a0cd (table rebalance_test_table_2 [id=facc024bfd644356a8879fd9661c928d]) successfully deleted
I20240501 14:05:36.622979 5384 raft_consensus.cc:1261] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:05:36.623057 8932 tablet_replica.cc:330] T ba6cba753e934e719c730593c0706346 P e9c60449cf3c46f8aca4ab6c535302a4: stopping tablet replica
I20240501 14:05:36.623308 8932 raft_consensus.cc:2229] T ba6cba753e934e719c730593c0706346 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:05:36.623576 8932 raft_consensus.cc:2258] T ba6cba753e934e719c730593c0706346 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:05:36.623762 8500 consensus_queue.cc:1035] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:05:36.628298 7025 raft_consensus.cc:1261] T e7d5f02ccba34254a4d20951878b1214 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:05:36.631525 8932 ts_tablet_manager.cc:1875] T ba6cba753e934e719c730593c0706346 P e9c60449cf3c46f8aca4ab6c535302a4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:05:36.633911 5384 raft_consensus.cc:2932] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:36.637225 4973 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:36.637681 4992 raft_consensus.cc:1261] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:36.638339 8905 consensus_queue.cc:1035] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:36.498390 8590 consensus_queue.cc:260] T 43b46fcf827c4275a47291afb107c290 P 19a6d2d8a08142d5bffcde3a37234692 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } }
I20240501 14:05:36.646422 8590 ts_tablet_manager.cc:1398] T 43b46fcf827c4275a47291afb107c290 P 19a6d2d8a08142d5bffcde3a37234692: Time spent starting tablet: real 0.149s user 0.005s sys 0.000s
I20240501 14:05:36.651990 5146 tablet_copy_service.cc:342] P e9c60449cf3c46f8aca4ab6c535302a4: Request end of tablet copy session 19a6d2d8a08142d5bffcde3a37234692-43b46fcf827c4275a47291afb107c290 received from {username='slave'} at 127.29.124.134:41917
I20240501 14:05:36.652113 5146 tablet_copy_service.cc:433] P e9c60449cf3c46f8aca4ab6c535302a4: ending tablet copy session 19a6d2d8a08142d5bffcde3a37234692-43b46fcf827c4275a47291afb107c290 on tablet 43b46fcf827c4275a47291afb107c290 with peer 19a6d2d8a08142d5bffcde3a37234692
I20240501 14:05:36.654166 8891 consensus_queue.cc:1035] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:05:36.657608 4852 catalog_manager.cc:5484] T 51cbcf4888c440c2ad096c988ab2d968 P 938a24b0e6a040b7966225af26558ff8 reported cstate change: config changed from index 7 to 8, VOTER 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129) evicted. New cstate: current_term: 1 leader_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:36.659294 6888 raft_consensus.cc:1261] T e7d5f02ccba34254a4d20951878b1214 P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:05:36.660380 8804 consensus_queue.cc:1035] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:05:36.660946 8891 raft_consensus.cc:2932] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:36.665567 4973 tablet_service.cc:1514] Processing DeleteTablet for tablet 51cbcf4888c440c2ad096c988ab2d968 with delete_type TABLET_DATA_TOMBSTONED (TS 04088ee3bea440a7bd5ff772152b9410 not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:41194
I20240501 14:05:36.666889 4839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e7d5f02ccba34254a4d20951878b1214 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:05:36.667202 7024 raft_consensus.cc:2932] T e7d5f02ccba34254a4d20951878b1214 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:36.668334 4846 catalog_manager.cc:5484] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf reported cstate change: config changed from index 9 to 10, VOTER 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129) evicted. New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:05:36.669612 7287 raft_consensus.cc:2932] T e7d5f02ccba34254a4d20951878b1214 P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:36.676911 6892 raft_consensus.cc:858] T e7d5f02ccba34254a4d20951878b1214 P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }.
Pending config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }:
New config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
W20240501 14:05:36.677031 6892 raft_consensus.cc:1565] T e7d5f02ccba34254a4d20951878b1214 P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Could not prepare op 'term: 1 index: 10' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }.
Pending config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }:
New config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:05:36.677423 5255 raft_consensus.cc:2932] T e7d5f02ccba34254a4d20951878b1214 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:36.677769 6892 raft_consensus.cc:2932] T e7d5f02ccba34254a4d20951878b1214 P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:36.696496 8804 raft_consensus.cc:1050] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf: attempting to promote NON_VOTER 6dac90f01e744f9c8701a404440a4c77 to VOTER
I20240501 14:05:36.697368 8804 consensus_queue.cc:237] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:05:36.699007 7285 raft_consensus.cc:2932] T 61a274b24f53458f88c2cde4f8dbe4f4 P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:36.703498 5254 consensus_queue.cc:237] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:05:36.705945 4974 tablet_service.cc:1514] Processing DeleteTablet for tablet e7d5f02ccba34254a4d20951878b1214 with delete_type TABLET_DATA_TOMBSTONED (TS 04088ee3bea440a7bd5ff772152b9410 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:41194
I20240501 14:05:36.709908 6888 raft_consensus.cc:1261] T 71660575d94748d8a9329f909a42a0cd P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:05:36.710510 8905 consensus_queue.cc:1035] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:05:36.712723 8905 raft_consensus.cc:2932] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:36.713785 6888 raft_consensus.cc:2932] T 71660575d94748d8a9329f909a42a0cd P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:36.716907 6889 raft_consensus.cc:1261] T 61a274b24f53458f88c2cde4f8dbe4f4 P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:05:36.717454 8352 consensus_queue.cc:1035] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:05:36.719822 4848 catalog_manager.cc:5484] T 71660575d94748d8a9329f909a42a0cd P 938a24b0e6a040b7966225af26558ff8 reported cstate change: config changed from index 8 to 9, VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) evicted. New cstate: current_term: 1 leader_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:36.721383 7024 raft_consensus.cc:1261] T 61a274b24f53458f88c2cde4f8dbe4f4 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:05:36.721603 8950 tablet_replica.cc:330] T 51cbcf4888c440c2ad096c988ab2d968 P 04088ee3bea440a7bd5ff772152b9410: stopping tablet replica
I20240501 14:05:36.721702 4838 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 71660575d94748d8a9329f909a42a0cd with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:05:36.721797 8950 raft_consensus.cc:2229] T 51cbcf4888c440c2ad096c988ab2d968 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:05:36.721850 8352 consensus_queue.cc:1035] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:05:36.722133 8950 raft_consensus.cc:2258] T 51cbcf4888c440c2ad096c988ab2d968 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:05:36.723285 4992 raft_consensus.cc:1261] T 71660575d94748d8a9329f909a42a0cd P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:05:36.723484 7004 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:36.723798 8905 consensus_queue.cc:1035] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:05:36.724655 7267 tablet_service.cc:1514] Processing DeleteTablet for tablet 71660575d94748d8a9329f909a42a0cd with delete_type TABLET_DATA_TOMBSTONED (TS 6dac90f01e744f9c8701a404440a4c77 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:52244
I20240501 14:05:36.725606 4995 raft_consensus.cc:2932] T 71660575d94748d8a9329f909a42a0cd P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:36.724303 8891 raft_consensus.cc:2932] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:36.730633 7284 raft_consensus.cc:1203] T 2c39b53867e047faaf24cc6f87a8100c P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240501 14:05:36.732973 8950 ts_tablet_manager.cc:1875] T 51cbcf4888c440c2ad096c988ab2d968 P 04088ee3bea440a7bd5ff772152b9410: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:05:36.734323 7025 raft_consensus.cc:2932] T 61a274b24f53458f88c2cde4f8dbe4f4 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:36.739921 4848 catalog_manager.cc:5484] T 61a274b24f53458f88c2cde4f8dbe4f4 P 19a6d2d8a08142d5bffcde3a37234692 reported cstate change: config changed from index 9 to 10, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:36.740674 5126 raft_consensus.cc:1261] T 61a274b24f53458f88c2cde4f8dbe4f4 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:05:36.741168 8612 consensus_queue.cc:1035] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:05:36.742540 5125 raft_consensus.cc:2932] T 61a274b24f53458f88c2cde4f8dbe4f4 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:36.749912 7136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:36.752983 6888 raft_consensus.cc:2932] T 61a274b24f53458f88c2cde4f8dbe4f4 P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:36.753615 7286 raft_consensus.cc:1261] T 61a274b24f53458f88c2cde4f8dbe4f4 P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:05:36.754460 8890 consensus_queue.cc:1035] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:05:36.756316 7284 raft_consensus.cc:2932] T 61a274b24f53458f88c2cde4f8dbe4f4 P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:36.765847 7004 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:36.713205 8932 ts_tablet_manager.cc:1888] T ba6cba753e934e719c730593c0706346 P e9c60449cf3c46f8aca4ab6c535302a4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:05:36.767375 8932 log.cc:1196] T ba6cba753e934e719c730593c0706346 P e9c60449cf3c46f8aca4ab6c535302a4: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/wal/wals/ba6cba753e934e719c730593c0706346
I20240501 14:05:36.770586 4839 catalog_manager.cc:4830] TS e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377): tablet ba6cba753e934e719c730593c0706346 (table rebalance_test_table_4 [id=84aeba643ddd44cfbbd6a34f81703482]) successfully deleted
I20240501 14:05:36.771786 8932 tablet_replica.cc:330] T ec1e81011c6a46bc8a39aecf3a5e71e4 P e9c60449cf3c46f8aca4ab6c535302a4: stopping tablet replica
I20240501 14:05:36.772025 8932 raft_consensus.cc:2229] T ec1e81011c6a46bc8a39aecf3a5e71e4 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:05:36.772296 8932 raft_consensus.cc:2258] T ec1e81011c6a46bc8a39aecf3a5e71e4 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:05:36.774171 8932 ts_tablet_manager.cc:1875] T ec1e81011c6a46bc8a39aecf3a5e71e4 P e9c60449cf3c46f8aca4ab6c535302a4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:05:36.791563 8953 tablet_replica.cc:330] T 71660575d94748d8a9329f909a42a0cd P 6dac90f01e744f9c8701a404440a4c77: stopping tablet replica
I20240501 14:05:36.791765 8953 raft_consensus.cc:2229] T 71660575d94748d8a9329f909a42a0cd P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:05:36.792120 8953 raft_consensus.cc:2258] T 71660575d94748d8a9329f909a42a0cd P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:05:36.806839 8925 tablet_copy_client.cc:308] T de6c7753c8564107b858876fcfa5924d P 938a24b0e6a040b7966225af26558ff8: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.131:33209
I20240501 14:05:36.807662 5276 tablet_copy_service.cc:140] P 1c3fdc5272484ac09ef3e5e36a08f27c: Received BeginTabletCopySession request for tablet de6c7753c8564107b858876fcfa5924d from peer 938a24b0e6a040b7966225af26558ff8 ({username='slave'} at 127.29.124.133:49097)
I20240501 14:05:36.807817 5276 tablet_copy_service.cc:161] P 1c3fdc5272484ac09ef3e5e36a08f27c: Beginning new tablet copy session on tablet de6c7753c8564107b858876fcfa5924d from peer 938a24b0e6a040b7966225af26558ff8 at {username='slave'} at 127.29.124.133:49097: session id = 938a24b0e6a040b7966225af26558ff8-de6c7753c8564107b858876fcfa5924d
I20240501 14:05:36.810525 5276 tablet_copy_source_session.cc:205] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:05:36.812825 8925 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de6c7753c8564107b858876fcfa5924d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:36.783149 4993 raft_consensus.cc:2932] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:36.820751 5236 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:36.822741 8932 ts_tablet_manager.cc:1888] T ec1e81011c6a46bc8a39aecf3a5e71e4 P e9c60449cf3c46f8aca4ab6c535302a4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:05:36.822870 8932 log.cc:1196] T ec1e81011c6a46bc8a39aecf3a5e71e4 P e9c60449cf3c46f8aca4ab6c535302a4: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/wal/wals/ec1e81011c6a46bc8a39aecf3a5e71e4
I20240501 14:05:36.823565 4839 catalog_manager.cc:4830] TS e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377): tablet ec1e81011c6a46bc8a39aecf3a5e71e4 (table rebalance_test_table_9 [id=5fc33bdfba464f65b68de5887d425ab7]) successfully deleted
I20240501 14:05:36.824957 8953 ts_tablet_manager.cc:1875] T 71660575d94748d8a9329f909a42a0cd P 6dac90f01e744f9c8701a404440a4c77: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:05:36.826896 5382 consensus_queue.cc:237] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } }
I20240501 14:05:36.829895 5255 raft_consensus.cc:1261] T e7d5f02ccba34254a4d20951878b1214 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:05:36.831539 8247 consensus_queue.cc:1035] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:05:36.840198 4973 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:36.842733 7025 raft_consensus.cc:1261] T e7d5f02ccba34254a4d20951878b1214 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:05:36.843588 8247 consensus_queue.cc:1035] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:05:36.843974 7287 raft_consensus.cc:1261] T e7d5f02ccba34254a4d20951878b1214 P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:05:36.844509 8247 consensus_queue.cc:1035] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:05:36.857381 8247 raft_consensus.cc:2932] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:36.861301 4839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e7d5f02ccba34254a4d20951878b1214 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:05:36.862635 5255 raft_consensus.cc:2932] T e7d5f02ccba34254a4d20951878b1214 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:36.863039 6892 raft_consensus.cc:2932] T e7d5f02ccba34254a4d20951878b1214 P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, VOTER 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:36.870586 7285 raft_consensus.cc:2932] T e7d5f02ccba34254a4d20951878b1214 P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:36.871066 4848 catalog_manager.cc:5484] T e7d5f02ccba34254a4d20951878b1214 P 1c3fdc5272484ac09ef3e5e36a08f27c reported cstate change: config changed from index 10 to 11, VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) evicted. New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:36.873525 5365 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:36.878427 6872 tablet_service.cc:1514] Processing DeleteTablet for tablet e7d5f02ccba34254a4d20951878b1214 with delete_type TABLET_DATA_TOMBSTONED (TS 938a24b0e6a040b7966225af26558ff8 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:50548
I20240501 14:05:36.878854 7024 raft_consensus.cc:2932] T e7d5f02ccba34254a4d20951878b1214 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:36.891014 7267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:36.899855 8925 tablet_copy_client.cc:783] T de6c7753c8564107b858876fcfa5924d P 938a24b0e6a040b7966225af26558ff8: tablet copy: Starting download of 0 data blocks...
I20240501 14:05:36.912323 7267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:36.928123 8953 ts_tablet_manager.cc:1888] T 71660575d94748d8a9329f909a42a0cd P 6dac90f01e744f9c8701a404440a4c77: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 14:05:36.932113 8953 log.cc:1196] T 71660575d94748d8a9329f909a42a0cd P 6dac90f01e744f9c8701a404440a4c77: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-7/wal/wals/71660575d94748d8a9329f909a42a0cd
I20240501 14:05:36.933912 4841 catalog_manager.cc:4830] TS 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): tablet 71660575d94748d8a9329f909a42a0cd (table rebalance_test_table_2 [id=facc024bfd644356a8879fd9661c928d]) successfully deleted
I20240501 14:05:36.942737 5384 consensus_queue.cc:237] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:05:36.944612 5382 consensus_queue.cc:237] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } }
I20240501 14:05:36.944979 8950 ts_tablet_manager.cc:1888] T 51cbcf4888c440c2ad096c988ab2d968 P 04088ee3bea440a7bd5ff772152b9410: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240501 14:05:36.945132 8950 log.cc:1196] T 51cbcf4888c440c2ad096c988ab2d968 P 04088ee3bea440a7bd5ff772152b9410: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/wal/wals/51cbcf4888c440c2ad096c988ab2d968
I20240501 14:05:36.946061 7287 raft_consensus.cc:1261] T 61a274b24f53458f88c2cde4f8dbe4f4 P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:05:36.946058 4839 catalog_manager.cc:4830] TS 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129:44033): tablet 51cbcf4888c440c2ad096c988ab2d968 (table rebalance_test_table_3 [id=41272b857ab045629ca60ab2467097b0]) successfully deleted
I20240501 14:05:36.946523 8950 tablet_replica.cc:330] T e7d5f02ccba34254a4d20951878b1214 P 04088ee3bea440a7bd5ff772152b9410: stopping tablet replica
I20240501 14:05:36.946714 8950 raft_consensus.cc:2229] T e7d5f02ccba34254a4d20951878b1214 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:05:36.946812 8890 consensus_queue.cc:1035] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:05:36.947175 8950 raft_consensus.cc:2258] T e7d5f02ccba34254a4d20951878b1214 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:05:36.947501 7025 raft_consensus.cc:1261] T e7d5f02ccba34254a4d20951878b1214 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:05:36.948154 8890 consensus_queue.cc:1035] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:05:36.948376 7287 raft_consensus.cc:1261] T e7d5f02ccba34254a4d20951878b1214 P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:05:36.948890 8890 consensus_queue.cc:1035] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:05:36.950713 8352 raft_consensus.cc:2932] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:36.951366 7025 raft_consensus.cc:2932] T e7d5f02ccba34254a4d20951878b1214 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:36.957273 7025 raft_consensus.cc:1261] T 61a274b24f53458f88c2cde4f8dbe4f4 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:05:36.957556 4847 catalog_manager.cc:5484] T e7d5f02ccba34254a4d20951878b1214 P 19a6d2d8a08142d5bffcde3a37234692 reported cstate change: config changed from index 11 to 12, VOTER 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) evicted. New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:36.957840 8612 consensus_queue.cc:1035] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:05:36.958602 4839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e7d5f02ccba34254a4d20951878b1214 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:05:36.959360 7285 raft_consensus.cc:2932] T e7d5f02ccba34254a4d20951878b1214 P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:36.960817 8925 tablet_copy_client.cc:647] T de6c7753c8564107b858876fcfa5924d P 938a24b0e6a040b7966225af26558ff8: tablet copy: Starting download of 1 WAL segments...
I20240501 14:05:36.924731 8980 tablet_replica.cc:330] T e7d5f02ccba34254a4d20951878b1214 P 938a24b0e6a040b7966225af26558ff8: stopping tablet replica
I20240501 14:05:36.963582 8503 raft_consensus.cc:2932] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:36.966154 8980 raft_consensus.cc:2229] T e7d5f02ccba34254a4d20951878b1214 P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:05:36.966625 8980 raft_consensus.cc:2258] T e7d5f02ccba34254a4d20951878b1214 P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:05:36.967322 4839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 61a274b24f53458f88c2cde4f8dbe4f4 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:05:36.969508 8980 ts_tablet_manager.cc:1875] T e7d5f02ccba34254a4d20951878b1214 P 938a24b0e6a040b7966225af26558ff8: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:05:36.971805 7025 raft_consensus.cc:2932] T 61a274b24f53458f88c2cde4f8dbe4f4 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:36.976334 7285 raft_consensus.cc:2932] T 61a274b24f53458f88c2cde4f8dbe4f4 P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:36.976637 4854 catalog_manager.cc:5484] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf reported cstate change: config changed from index 10 to 11, VOTER e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130) evicted. New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:05:36.980418 5106 tablet_service.cc:1514] Processing DeleteTablet for tablet 61a274b24f53458f88c2cde4f8dbe4f4 with delete_type TABLET_DATA_TOMBSTONED (TS e9c60449cf3c46f8aca4ab6c535302a4 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:41850
I20240501 14:05:36.980743 8932 tablet_replica.cc:330] T 61a274b24f53458f88c2cde4f8dbe4f4 P e9c60449cf3c46f8aca4ab6c535302a4: stopping tablet replica
I20240501 14:05:36.980929 8932 raft_consensus.cc:2229] T 61a274b24f53458f88c2cde4f8dbe4f4 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:05:36.981472 8932 raft_consensus.cc:2258] T 61a274b24f53458f88c2cde4f8dbe4f4 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:05:36.982482 8932 ts_tablet_manager.cc:1875] T 61a274b24f53458f88c2cde4f8dbe4f4 P e9c60449cf3c46f8aca4ab6c535302a4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:05:36.984395 6872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:36.989679 7136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:36.991089 6892 raft_consensus.cc:1261] T 61a274b24f53458f88c2cde4f8dbe4f4 P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:05:36.991830 8891 consensus_queue.cc:1035] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:05:36.998986 4973 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:36.999666 8980 ts_tablet_manager.cc:1888] T e7d5f02ccba34254a4d20951878b1214 P 938a24b0e6a040b7966225af26558ff8: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240501 14:05:37.000254 7004 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:37.005578 8980 log.cc:1196] T e7d5f02ccba34254a4d20951878b1214 P 938a24b0e6a040b7966225af26558ff8: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-4/wal/wals/e7d5f02ccba34254a4d20951878b1214
I20240501 14:05:37.010890 8932 ts_tablet_manager.cc:1888] T 61a274b24f53458f88c2cde4f8dbe4f4 P e9c60449cf3c46f8aca4ab6c535302a4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240501 14:05:37.011088 8932 log.cc:1196] T 61a274b24f53458f88c2cde4f8dbe4f4 P e9c60449cf3c46f8aca4ab6c535302a4: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/wal/wals/61a274b24f53458f88c2cde4f8dbe4f4
I20240501 14:05:37.011931 4840 catalog_manager.cc:4830] TS 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): tablet e7d5f02ccba34254a4d20951878b1214 (table rebalance_test_table_4 [id=84aeba643ddd44cfbbd6a34f81703482]) successfully deleted
I20240501 14:05:37.015795 6892 raft_consensus.cc:2932] T 61a274b24f53458f88c2cde4f8dbe4f4 P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:37.025123 5236 tablet_service.cc:1514] Processing DeleteTablet for tablet e7d5f02ccba34254a4d20951878b1214 with delete_type TABLET_DATA_TOMBSTONED (TS 1c3fdc5272484ac09ef3e5e36a08f27c not found in new config with opid_index 12) from {username='slave'} at 127.0.0.1:39020
I20240501 14:05:37.037637 8897 raft_consensus.cc:1050] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c: attempting to promote NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 to VOTER
I20240501 14:05:37.037734 6872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:37.038468 8897 consensus_queue.cc:237] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } }
I20240501 14:05:37.039173 5365 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:37.044538 5106 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:37.053790 7136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:37.056762 8925 tablet_copy_client.cc:523] T de6c7753c8564107b858876fcfa5924d P 938a24b0e6a040b7966225af26558ff8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:05:37.060899 8925 tablet_bootstrap.cc:492] T de6c7753c8564107b858876fcfa5924d P 938a24b0e6a040b7966225af26558ff8: Bootstrap starting.
I20240501 14:05:37.060923 4839 catalog_manager.cc:4830] TS e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377): tablet 61a274b24f53458f88c2cde4f8dbe4f4 (table rebalance_test_table_9 [id=5fc33bdfba464f65b68de5887d425ab7]) successfully deleted
I20240501 14:05:37.063560 7024 raft_consensus.cc:1261] T 404e2851055541f0be1bb0971a7cc72a P 19a6d2d8a08142d5bffcde3a37234692 [term 1 LEARNER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:37.066008 5234 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:37.066035 8897 consensus_queue.cc:1035] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:37.070539 5384 raft_consensus.cc:1261] T 404e2851055541f0be1bb0971a7cc72a P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:37.072830 8905 consensus_queue.cc:1035] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:37.078766 5234 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:37.081780 5125 raft_consensus.cc:1261] T 404e2851055541f0be1bb0971a7cc72a P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:37.082356 8905 consensus_queue.cc:1035] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:37.086532 7025 raft_consensus.cc:1203] T 43b46fcf827c4275a47291afb107c290 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:05:37.094080 8950 ts_tablet_manager.cc:1875] T e7d5f02ccba34254a4d20951878b1214 P 04088ee3bea440a7bd5ff772152b9410: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:05:37.106680 5106 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:37.143362 8950 ts_tablet_manager.cc:1888] T e7d5f02ccba34254a4d20951878b1214 P 04088ee3bea440a7bd5ff772152b9410: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.9
I20240501 14:05:37.147805 7267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:37.170131 6872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:37.178764 8897 raft_consensus.cc:2932] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:37.182590 5125 raft_consensus.cc:2932] T 404e2851055541f0be1bb0971a7cc72a P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:37.188088 4847 catalog_manager.cc:5484] T 404e2851055541f0be1bb0971a7cc72a P e9c60449cf3c46f8aca4ab6c535302a4 reported cstate change: config changed from index 4 to 5, 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:37.191890 5384 raft_consensus.cc:2932] T 404e2851055541f0be1bb0971a7cc72a P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:37.191062 8994 tablet_replica.cc:330] T e7d5f02ccba34254a4d20951878b1214 P 1c3fdc5272484ac09ef3e5e36a08f27c: stopping tablet replica
I20240501 14:05:37.192543 8994 raft_consensus.cc:2229] T e7d5f02ccba34254a4d20951878b1214 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:05:37.193215 8994 raft_consensus.cc:2258] T e7d5f02ccba34254a4d20951878b1214 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:05:37.195863 5365 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:37.206897 8874 tablet_bootstrap.cc:492] T 71bdb382b08341218c8bf2459fe65fc0 P 4360b59e3a154688a79ca5f9c0a37bc9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:05:37.207587 8874 tablet_bootstrap.cc:492] T 71bdb382b08341218c8bf2459fe65fc0 P 4360b59e3a154688a79ca5f9c0a37bc9: Bootstrap complete.
I20240501 14:05:37.208002 8874 ts_tablet_manager.cc:1367] T 71bdb382b08341218c8bf2459fe65fc0 P 4360b59e3a154688a79ca5f9c0a37bc9: Time spent bootstrapping tablet: real 1.024s user 0.047s sys 0.003s
I20240501 14:05:37.208997 8874 raft_consensus.cc:348] T 71bdb382b08341218c8bf2459fe65fc0 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } }
I20240501 14:05:37.209714 7024 raft_consensus.cc:2932] T 404e2851055541f0be1bb0971a7cc72a P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:37.211980 8950 log.cc:1196] T e7d5f02ccba34254a4d20951878b1214 P 04088ee3bea440a7bd5ff772152b9410: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/wal/wals/e7d5f02ccba34254a4d20951878b1214
I20240501 14:05:37.213016 4839 catalog_manager.cc:4830] TS 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129:44033): tablet e7d5f02ccba34254a4d20951878b1214 (table rebalance_test_table_4 [id=84aeba643ddd44cfbbd6a34f81703482]) successfully deleted
I20240501 14:05:37.215792 5251 consensus_queue.cc:237] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } }
I20240501 14:05:37.220132 5125 raft_consensus.cc:1261] T 404e2851055541f0be1bb0971a7cc72a P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:37.221019 8500 consensus_queue.cc:1035] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:37.223441 8500 raft_consensus.cc:2932] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER be56773105e744919601127a94e98adf (127.29.124.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:37.224783 8994 ts_tablet_manager.cc:1875] T e7d5f02ccba34254a4d20951878b1214 P 1c3fdc5272484ac09ef3e5e36a08f27c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:05:37.225979 5125 raft_consensus.cc:2932] T 404e2851055541f0be1bb0971a7cc72a P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER be56773105e744919601127a94e98adf (127.29.124.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:37.231794 4853 catalog_manager.cc:5484] T 404e2851055541f0be1bb0971a7cc72a P e9c60449cf3c46f8aca4ab6c535302a4 reported cstate change: config changed from index 5 to 6, VOTER be56773105e744919601127a94e98adf (127.29.124.132) evicted. New cstate: current_term: 1 leader_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:37.238888 5365 tablet_service.cc:1514] Processing DeleteTablet for tablet 404e2851055541f0be1bb0971a7cc72a with delete_type TABLET_DATA_TOMBSTONED (TS be56773105e744919601127a94e98adf not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:43020
I20240501 14:05:37.243785 8874 raft_consensus.cc:729] T 71bdb382b08341218c8bf2459fe65fc0 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4360b59e3a154688a79ca5f9c0a37bc9, State: Initialized, Role: LEARNER
I20240501 14:05:37.244102 8874 consensus_queue.cc:260] T 71bdb382b08341218c8bf2459fe65fc0 P 4360b59e3a154688a79ca5f9c0a37bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } }
I20240501 14:05:37.244948 4838 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 404e2851055541f0be1bb0971a7cc72a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:05:37.255612 7025 raft_consensus.cc:1261] T 404e2851055541f0be1bb0971a7cc72a P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:37.256558 8500 consensus_queue.cc:1035] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:37.257972 7025 raft_consensus.cc:2932] T 404e2851055541f0be1bb0971a7cc72a P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER be56773105e744919601127a94e98adf (127.29.124.132) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:37.288599 8874 ts_tablet_manager.cc:1398] T 71bdb382b08341218c8bf2459fe65fc0 P 4360b59e3a154688a79ca5f9c0a37bc9: Time spent starting tablet: real 0.080s user 0.002s sys 0.000s
I20240501 14:05:37.289479 5406 tablet_copy_service.cc:342] P be56773105e744919601127a94e98adf: Request end of tablet copy session 4360b59e3a154688a79ca5f9c0a37bc9-71bdb382b08341218c8bf2459fe65fc0 received from {username='slave'} at 127.29.124.135:51309
I20240501 14:05:37.289602 5406 tablet_copy_service.cc:433] P be56773105e744919601127a94e98adf: ending tablet copy session 4360b59e3a154688a79ca5f9c0a37bc9-71bdb382b08341218c8bf2459fe65fc0 on tablet 71bdb382b08341218c8bf2459fe65fc0 with peer 4360b59e3a154688a79ca5f9c0a37bc9
I20240501 14:05:37.302281 5106 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:37.325711 9010 tablet_replica.cc:330] T 404e2851055541f0be1bb0971a7cc72a P be56773105e744919601127a94e98adf: stopping tablet replica
I20240501 14:05:37.326028 9010 raft_consensus.cc:2229] T 404e2851055541f0be1bb0971a7cc72a P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:05:37.327049 9010 raft_consensus.cc:2258] T 404e2851055541f0be1bb0971a7cc72a P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:05:37.337800 9010 ts_tablet_manager.cc:1875] T 404e2851055541f0be1bb0971a7cc72a P be56773105e744919601127a94e98adf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:05:37.341974 8994 ts_tablet_manager.cc:1888] T e7d5f02ccba34254a4d20951878b1214 P 1c3fdc5272484ac09ef3e5e36a08f27c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.11
I20240501 14:05:37.344062 8994 log.cc:1196] T e7d5f02ccba34254a4d20951878b1214 P 1c3fdc5272484ac09ef3e5e36a08f27c: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/wal/wals/e7d5f02ccba34254a4d20951878b1214
I20240501 14:05:37.346426 4838 catalog_manager.cc:4830] TS 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209): tablet e7d5f02ccba34254a4d20951878b1214 (table rebalance_test_table_4 [id=84aeba643ddd44cfbbd6a34f81703482]) successfully deleted
I20240501 14:05:37.368968 8891 raft_consensus.cc:1050] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf: attempting to promote NON_VOTER 6dac90f01e744f9c8701a404440a4c77 to VOTER
I20240501 14:05:37.374533 8891 consensus_queue.cc:237] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } }
I20240501 14:05:37.389164 9010 ts_tablet_manager.cc:1888] T 404e2851055541f0be1bb0971a7cc72a P be56773105e744919601127a94e98adf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:05:37.389400 9010 log.cc:1196] T 404e2851055541f0be1bb0971a7cc72a P be56773105e744919601127a94e98adf: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/wal/wals/404e2851055541f0be1bb0971a7cc72a
I20240501 14:05:37.402089 4839 catalog_manager.cc:4830] TS be56773105e744919601127a94e98adf (127.29.124.132:35459): tablet 404e2851055541f0be1bb0971a7cc72a (table rebalance_test_table_3 [id=41272b857ab045629ca60ab2467097b0]) successfully deleted
I20240501 14:05:37.410293 7285 raft_consensus.cc:1261] T 2c39b53867e047faaf24cc6f87a8100c P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:05:37.410912 6892 raft_consensus.cc:1261] T 2c39b53867e047faaf24cc6f87a8100c P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:05:37.411926 8612 consensus_queue.cc:1035] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:05:37.413640 5125 raft_consensus.cc:1261] T 2c39b53867e047faaf24cc6f87a8100c P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:05:37.414609 8804 consensus_queue.cc:1035] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:05:37.415107 8804 consensus_queue.cc:1035] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:05:37.527138 7285 raft_consensus.cc:2932] T 2c39b53867e047faaf24cc6f87a8100c P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:37.543416 8804 raft_consensus.cc:2932] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:37.552428 5125 raft_consensus.cc:2932] T 2c39b53867e047faaf24cc6f87a8100c P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:37.559720 7155 raft_consensus.cc:1203] T 71bdb382b08341218c8bf2459fe65fc0 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:05:37.563046 6888 raft_consensus.cc:2932] T 2c39b53867e047faaf24cc6f87a8100c P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:37.564793 4853 catalog_manager.cc:5484] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf reported cstate change: config changed from index 9 to 10, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:05:37.570505 5384 consensus_queue.cc:237] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } }
I20240501 14:05:37.578512 4973 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:37.590476 5236 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:37.593006 7287 raft_consensus.cc:1261] T 2c39b53867e047faaf24cc6f87a8100c P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:05:37.593876 9030 consensus_queue.cc:1035] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:05:37.596525 8891 raft_consensus.cc:2932] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:37.601475 4839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2c39b53867e047faaf24cc6f87a8100c with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:05:37.603267 4847 catalog_manager.cc:5484] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf reported cstate change: config changed from index 10 to 11, VOTER e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130) evicted. New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:05:37.609658 5106 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c39b53867e047faaf24cc6f87a8100c with delete_type TABLET_DATA_TOMBSTONED (TS e9c60449cf3c46f8aca4ab6c535302a4 not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:41850
I20240501 14:05:37.613328 7287 raft_consensus.cc:2932] T 2c39b53867e047faaf24cc6f87a8100c P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:37.623095 9042 tablet_replica.cc:330] T 2c39b53867e047faaf24cc6f87a8100c P e9c60449cf3c46f8aca4ab6c535302a4: stopping tablet replica
I20240501 14:05:37.625214 9042 raft_consensus.cc:2229] T 2c39b53867e047faaf24cc6f87a8100c P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:05:37.626181 9042 raft_consensus.cc:2258] T 2c39b53867e047faaf24cc6f87a8100c P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:05:37.627766 9042 ts_tablet_manager.cc:1875] T 2c39b53867e047faaf24cc6f87a8100c P e9c60449cf3c46f8aca4ab6c535302a4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:05:37.628734 6889 raft_consensus.cc:1261] T 2c39b53867e047faaf24cc6f87a8100c P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:05:37.629828 8891 consensus_queue.cc:1035] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:05:37.634240 6889 raft_consensus.cc:2932] T 2c39b53867e047faaf24cc6f87a8100c P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:37.636276 7004 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:37.673895 6872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:37.682538 5365 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:37.738334 9042 ts_tablet_manager.cc:1888] T 2c39b53867e047faaf24cc6f87a8100c P e9c60449cf3c46f8aca4ab6c535302a4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240501 14:05:37.738608 9042 log.cc:1196] T 2c39b53867e047faaf24cc6f87a8100c P e9c60449cf3c46f8aca4ab6c535302a4: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/wal/wals/2c39b53867e047faaf24cc6f87a8100c
I20240501 14:05:37.739526 4839 catalog_manager.cc:4830] TS e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377): tablet 2c39b53867e047faaf24cc6f87a8100c (table rebalance_test_table_5 [id=3794b1ccb5004f7cb92a229ef6ce3595]) successfully deleted
I20240501 14:05:37.746001 5106 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:37.758138 7267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:37.770834 7136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:37.783190 8925 tablet_bootstrap.cc:492] T de6c7753c8564107b858876fcfa5924d P 938a24b0e6a040b7966225af26558ff8: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:05:37.783639 8925 tablet_bootstrap.cc:492] T de6c7753c8564107b858876fcfa5924d P 938a24b0e6a040b7966225af26558ff8: Bootstrap complete.
I20240501 14:05:37.783897 8925 ts_tablet_manager.cc:1367] T de6c7753c8564107b858876fcfa5924d P 938a24b0e6a040b7966225af26558ff8: Time spent bootstrapping tablet: real 0.723s user 0.047s sys 0.000s
I20240501 14:05:37.784660 8925 raft_consensus.cc:348] T de6c7753c8564107b858876fcfa5924d P 938a24b0e6a040b7966225af26558ff8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }
I20240501 14:05:37.784916 8925 raft_consensus.cc:729] T de6c7753c8564107b858876fcfa5924d P 938a24b0e6a040b7966225af26558ff8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 938a24b0e6a040b7966225af26558ff8, State: Initialized, Role: LEARNER
I20240501 14:05:37.801929 9051 raft_consensus.cc:1050] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4: attempting to promote NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 to VOTER
I20240501 14:05:37.802942 9051 consensus_queue.cc:237] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } }
I20240501 14:05:37.827755 4993 raft_consensus.cc:1261] T 43b46fcf827c4275a47291afb107c290 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:37.827939 7024 raft_consensus.cc:1261] T 43b46fcf827c4275a47291afb107c290 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 LEARNER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:37.829180 5384 raft_consensus.cc:1261] T 43b46fcf827c4275a47291afb107c290 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:37.829752 9048 consensus_queue.cc:1035] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:37.832152 9057 consensus_queue.cc:1035] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:37.832917 9051 consensus_queue.cc:1035] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:37.861479 9048 raft_consensus.cc:2932] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:37.866884 5384 raft_consensus.cc:2932] T 43b46fcf827c4275a47291afb107c290 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:37.870632 4853 catalog_manager.cc:5484] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 reported cstate change: config changed from index 4 to 5, 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:05:37.785118 8925 consensus_queue.cc:260] T de6c7753c8564107b858876fcfa5924d P 938a24b0e6a040b7966225af26558ff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }
I20240501 14:05:37.887826 8925 ts_tablet_manager.cc:1398] T de6c7753c8564107b858876fcfa5924d P 938a24b0e6a040b7966225af26558ff8: Time spent starting tablet: real 0.103s user 0.005s sys 0.000s
I20240501 14:05:37.874425 4993 raft_consensus.cc:2932] T 43b46fcf827c4275a47291afb107c290 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:37.893071 5276 tablet_copy_service.cc:342] P 1c3fdc5272484ac09ef3e5e36a08f27c: Request end of tablet copy session 938a24b0e6a040b7966225af26558ff8-de6c7753c8564107b858876fcfa5924d received from {username='slave'} at 127.29.124.133:49097
I20240501 14:05:37.893375 5276 tablet_copy_service.cc:433] P 1c3fdc5272484ac09ef3e5e36a08f27c: ending tablet copy session 938a24b0e6a040b7966225af26558ff8-de6c7753c8564107b858876fcfa5924d on tablet de6c7753c8564107b858876fcfa5924d with peer 938a24b0e6a040b7966225af26558ff8
I20240501 14:05:37.899528 7024 raft_consensus.cc:2932] T 43b46fcf827c4275a47291afb107c290 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:37.909582 5125 consensus_queue.cc:237] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } }
I20240501 14:05:37.912340 5384 raft_consensus.cc:1261] T 43b46fcf827c4275a47291afb107c290 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:37.913004 9057 consensus_queue.cc:1035] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:37.916105 9048 raft_consensus.cc:2932] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:37.927924 6889 raft_consensus.cc:1203] T de6c7753c8564107b858876fcfa5924d P 938a24b0e6a040b7966225af26558ff8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:05:37.929250 4839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 43b46fcf827c4275a47291afb107c290 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:05:37.929472 5384 raft_consensus.cc:2932] T 43b46fcf827c4275a47291afb107c290 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:37.931510 4847 catalog_manager.cc:5484] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 reported cstate change: config changed from index 5 to 6, VOTER 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129) evicted. New cstate: current_term: 1 leader_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:05:37.942601 4973 tablet_service.cc:1514] Processing DeleteTablet for tablet 43b46fcf827c4275a47291afb107c290 with delete_type TABLET_DATA_TOMBSTONED (TS 04088ee3bea440a7bd5ff772152b9410 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41194
I20240501 14:05:37.943666 7025 raft_consensus.cc:1261] T 43b46fcf827c4275a47291afb107c290 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:37.946557 9051 consensus_queue.cc:1035] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:37.948239 7025 raft_consensus.cc:2932] T 43b46fcf827c4275a47291afb107c290 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:37.980947 9068 tablet_replica.cc:330] T 43b46fcf827c4275a47291afb107c290 P 04088ee3bea440a7bd5ff772152b9410: stopping tablet replica
I20240501 14:05:37.982482 9068 raft_consensus.cc:2229] T 43b46fcf827c4275a47291afb107c290 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:05:37.984741 9068 raft_consensus.cc:2258] T 43b46fcf827c4275a47291afb107c290 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:05:37.990628 9068 ts_tablet_manager.cc:1875] T 43b46fcf827c4275a47291afb107c290 P 04088ee3bea440a7bd5ff772152b9410: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:05:37.995160 4973 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:38.002537 7004 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:38.010593 8804 raft_consensus.cc:1050] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf: attempting to promote NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 to VOTER
I20240501 14:05:38.011253 8804 consensus_queue.cc:237] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } }
I20240501 14:05:38.016566 7136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:38.017426 7267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:38.038003 5236 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:38.043920 7155 raft_consensus.cc:1261] T 71bdb382b08341218c8bf2459fe65fc0 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 LEARNER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:38.044667 9030 consensus_queue.cc:1035] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:38.049747 5251 raft_consensus.cc:1261] T 71bdb382b08341218c8bf2459fe65fc0 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:38.050422 9030 consensus_queue.cc:1035] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:38.067572 4993 raft_consensus.cc:1261] T 71bdb382b08341218c8bf2459fe65fc0 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:38.068223 9030 consensus_queue.cc:1035] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:38.076757 5365 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:38.096560 6872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:38.110983 8804 raft_consensus.cc:2932] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:38.111728 5251 raft_consensus.cc:2932] T 71bdb382b08341218c8bf2459fe65fc0 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:38.116634 9068 ts_tablet_manager.cc:1888] T 43b46fcf827c4275a47291afb107c290 P 04088ee3bea440a7bd5ff772152b9410: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:05:38.122152 4993 consensus_queue.cc:237] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } }
I20240501 14:05:38.125083 4995 raft_consensus.cc:2932] T 71bdb382b08341218c8bf2459fe65fc0 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:38.129419 5125 consensus_queue.cc:237] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } }
I20240501 14:05:38.136853 4853 catalog_manager.cc:5484] T 71bdb382b08341218c8bf2459fe65fc0 P 04088ee3bea440a7bd5ff772152b9410 reported cstate change: config changed from index 4 to 5, 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:38.144584 7155 raft_consensus.cc:2932] T 71bdb382b08341218c8bf2459fe65fc0 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
W20240501 14:05:38.145774 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c8a961e711114f369ba06840c062b982. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:38.145795 4995 raft_consensus.cc:1261] T c8a961e711114f369ba06840c062b982 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:05:38.146293 9057 consensus_queue.cc:1035] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:05:38.147168 5384 consensus_queue.cc:237] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } }
I20240501 14:05:38.149907 4995 raft_consensus.cc:1261] T 71bdb382b08341218c8bf2459fe65fc0 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:38.151410 8612 consensus_queue.cc:1035] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:38.154816 5255 raft_consensus.cc:1261] T c8a961e711114f369ba06840c062b982 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:05:38.157373 9057 consensus_queue.cc:1035] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:05:38.165155 9030 raft_consensus.cc:2932] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:38.168196 4994 raft_consensus.cc:2932] T 71bdb382b08341218c8bf2459fe65fc0 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:38.169720 4839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 71bdb382b08341218c8bf2459fe65fc0 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:05:38.172641 4847 catalog_manager.cc:5484] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf reported cstate change: config changed from index 5 to 6, VOTER 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) evicted. New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:05:38.179677 5236 tablet_service.cc:1514] Processing DeleteTablet for tablet 71bdb382b08341218c8bf2459fe65fc0 with delete_type TABLET_DATA_TOMBSTONED (TS 1c3fdc5272484ac09ef3e5e36a08f27c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39020
I20240501 14:05:38.181692 9057 raft_consensus.cc:2932] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:38.189601 5255 raft_consensus.cc:2932] T c8a961e711114f369ba06840c062b982 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:38.190241 4853 catalog_manager.cc:5484] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 reported cstate change: config changed from index -1 to 4, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) added. New cstate: current_term: 1 leader_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:05:38.202075 5106 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:38.209619 4994 raft_consensus.cc:2932] T c8a961e711114f369ba06840c062b982 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
W20240501 14:05:38.210933 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 99d958617b7e4ef385b94f4c73f4dd84. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:38.216723 5384 raft_consensus.cc:1261] T 99d958617b7e4ef385b94f4c73f4dd84 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:05:38.217521 8375 consensus_queue.cc:1035] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:05:38.217775 9068 log.cc:1196] T 43b46fcf827c4275a47291afb107c290 P 04088ee3bea440a7bd5ff772152b9410: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/wal/wals/43b46fcf827c4275a47291afb107c290
I20240501 14:05:38.218540 4839 catalog_manager.cc:4830] TS 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129:44033): tablet 43b46fcf827c4275a47291afb107c290 (table rebalance_test_table_0 [id=7af67fe405db4bffbc53cb288b8bb715]) successfully deleted
I20240501 14:05:38.227744 9103 tablet_replica.cc:330] T 71bdb382b08341218c8bf2459fe65fc0 P 1c3fdc5272484ac09ef3e5e36a08f27c: stopping tablet replica
I20240501 14:05:38.228286 9103 raft_consensus.cc:2229] T 71bdb382b08341218c8bf2459fe65fc0 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:05:38.232656 7156 raft_consensus.cc:1261] T 71bdb382b08341218c8bf2459fe65fc0 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:38.233355 8804 consensus_queue.cc:1035] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:38.234858 7155 raft_consensus.cc:2932] T 71bdb382b08341218c8bf2459fe65fc0 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:38.239310 8375 raft_consensus.cc:2932] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:38.242115 5382 raft_consensus.cc:2932] T 99d958617b7e4ef385b94f4c73f4dd84 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:38.248868 9103 raft_consensus.cc:2258] T 71bdb382b08341218c8bf2459fe65fc0 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:05:38.236327 5251 raft_consensus.cc:1261] T 99d958617b7e4ef385b94f4c73f4dd84 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:05:38.252060 4853 catalog_manager.cc:5484] T 99d958617b7e4ef385b94f4c73f4dd84 P be56773105e744919601127a94e98adf reported cstate change: config changed from index -1 to 4, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) added. New cstate: current_term: 1 leader_uuid: "04088ee3bea440a7bd5ff772152b9410" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:38.252557 8375 consensus_queue.cc:1035] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:05:38.268556 5251 raft_consensus.cc:2932] T 99d958617b7e4ef385b94f4c73f4dd84 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:38.272461 9103 ts_tablet_manager.cc:1875] T 71bdb382b08341218c8bf2459fe65fc0 P 1c3fdc5272484ac09ef3e5e36a08f27c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:05:38.279528 9103 ts_tablet_manager.cc:1888] T 71bdb382b08341218c8bf2459fe65fc0 P 1c3fdc5272484ac09ef3e5e36a08f27c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:05:38.291618 9103 log.cc:1196] T 71bdb382b08341218c8bf2459fe65fc0 P 1c3fdc5272484ac09ef3e5e36a08f27c: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/wal/wals/71bdb382b08341218c8bf2459fe65fc0
I20240501 14:05:38.292629 4838 catalog_manager.cc:4830] TS 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209): tablet 71bdb382b08341218c8bf2459fe65fc0 (table rebalance_test_table_7 [id=763c884aecd640b49c65c692fbde9422]) successfully deleted
I20240501 14:05:38.295528 5251 consensus_queue.cc:237] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } }
I20240501 14:05:38.303851 7025 raft_consensus.cc:1261] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:05:38.304021 5125 raft_consensus.cc:1261] T 23ca2bf1e59f44fcac8a48aff8d056b1 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:05:38.304602 8897 consensus_queue.cc:1035] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 14:05:38.304929 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 23ca2bf1e59f44fcac8a48aff8d056b1. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:38.305104 8277 consensus_queue.cc:1035] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:05:38.305809 5384 consensus_queue.cc:237] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }
I20240501 14:05:38.333613 5126 raft_consensus.cc:1261] T 420642af520d45ffbc7f2d35f93e75a7 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240501 14:05:38.334542 5321 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 420642af520d45ffbc7f2d35f93e75a7. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:38.334854 5251 raft_consensus.cc:1261] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:05:38.334868 8804 consensus_queue.cc:1035] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:05:38.335548 8897 raft_consensus.cc:2932] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:38.336099 5125 raft_consensus.cc:2932] T 23ca2bf1e59f44fcac8a48aff8d056b1 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:38.337513 8804 consensus_queue.cc:1035] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:05:38.339720 7024 raft_consensus.cc:2932] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:38.345095 4847 catalog_manager.cc:5484] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 19a6d2d8a08142d5bffcde3a37234692 reported cstate change: config changed from index 6 to 7, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New cstate: current_term: 1 leader_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:38.377424 8804 raft_consensus.cc:2932] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:38.381669 5125 raft_consensus.cc:2932] T 420642af520d45ffbc7f2d35f93e75a7 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:38.387946 4853 catalog_manager.cc:5484] T 420642af520d45ffbc7f2d35f93e75a7 P e9c60449cf3c46f8aca4ab6c535302a4 reported cstate change: config changed from index -1 to 4, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:38.393306 5255 raft_consensus.cc:2932] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:38.409722 4992 consensus_queue.cc:237] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } }
I20240501 14:05:38.419579 7155 raft_consensus.cc:1261] T fee16905387d428aa02cd2070c86c790 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:05:38.420387 5126 raft_consensus.cc:1261] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
W20240501 14:05:38.421692 4930 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fee16905387d428aa02cd2070c86c790. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:38.421929 8375 consensus_queue.cc:1035] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:05:38.424105 8375 consensus_queue.cc:1035] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:05:38.430042 8375 raft_consensus.cc:2932] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } }
I20240501 14:05:38.433269 5126 raft_consensus.cc:2932] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } }
I20240501 14:05:38.433995 7155 raft_consensus.cc:2932] T fee16905387d428aa02cd2070c86c790 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } }
I20240501 14:05:38.442013 4854 catalog_manager.cc:5484] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 reported cstate change: config changed from index 10 to 11, NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) added. New cstate: current_term: 1 leader_uuid: "04088ee3bea440a7bd5ff772152b9410" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:05:38.456943 5384 consensus_queue.cc:237] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } }
I20240501 14:05:38.457549 5126 consensus_queue.cc:237] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } }
I20240501 14:05:38.459800 5382 raft_consensus.cc:1261] T 43b46fcf827c4275a47291afb107c290 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:05:38.460124 7025 raft_consensus.cc:1261] T 43b46fcf827c4275a47291afb107c290 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:05:38.460381 9048 consensus_queue.cc:1035] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:05:38.461297 9048 consensus_queue.cc:1035] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 14:05:38.463847 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 43b46fcf827c4275a47291afb107c290. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:38.463985 9048 raft_consensus.cc:2932] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
W20240501 14:05:38.464490 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f4d2b8c005b64bbb8e6621da068d15e5. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:38.464771 5251 raft_consensus.cc:1261] T f4d2b8c005b64bbb8e6621da068d15e5 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:05:38.465342 9030 consensus_queue.cc:1035] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:05:38.468278 7025 raft_consensus.cc:1261] T f4d2b8c005b64bbb8e6621da068d15e5 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:05:38.468921 9030 consensus_queue.cc:1035] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:05:38.468884 5384 raft_consensus.cc:2932] T 43b46fcf827c4275a47291afb107c290 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:38.470232 7024 raft_consensus.cc:2932] T 43b46fcf827c4275a47291afb107c290 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:38.474756 8891 raft_consensus.cc:2932] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:38.475363 4846 catalog_manager.cc:5484] T 43b46fcf827c4275a47291afb107c290 P be56773105e744919601127a94e98adf reported cstate change: config changed from index 6 to 7, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New cstate: current_term: 1 leader_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:38.475353 5251 raft_consensus.cc:2932] T f4d2b8c005b64bbb8e6621da068d15e5 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:38.484521 4854 catalog_manager.cc:5484] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf reported cstate change: config changed from index 6 to 7, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) added. New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:05:38.490164 5126 consensus_queue.cc:237] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }
I20240501 14:05:38.492645 5255 raft_consensus.cc:1261] T c8a961e711114f369ba06840c062b982 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:38.492995 4992 raft_consensus.cc:1261] T c8a961e711114f369ba06840c062b982 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:38.493292 9051 consensus_queue.cc:1035] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:38.493757 9051 consensus_queue.cc:1035] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:38.494028 7025 raft_consensus.cc:2932] T f4d2b8c005b64bbb8e6621da068d15e5 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
W20240501 14:05:38.496790 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c8a961e711114f369ba06840c062b982. This is attempt 1: this message will repeat every 5th retry.
W20240501 14:05:38.496973 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c8a961e711114f369ba06840c062b982. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:38.497536 9051 raft_consensus.cc:2932] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:38.498670 8500 raft_consensus.cc:1050] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c: attempting to promote NON_VOTER 938a24b0e6a040b7966225af26558ff8 to VOTER
I20240501 14:05:38.499413 8500 consensus_queue.cc:237] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:05:38.502063 5255 raft_consensus.cc:2932] T c8a961e711114f369ba06840c062b982 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:38.502286 4992 raft_consensus.cc:2932] T c8a961e711114f369ba06840c062b982 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:38.505626 4993 consensus_queue.cc:237] T a726769d24284fedadd8e9e63cd27d05 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } }
I20240501 14:05:38.512213 4846 catalog_manager.cc:5484] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 reported cstate change: config changed from index 4 to 5, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New cstate: current_term: 1 leader_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 14:05:38.534276 4930 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a726769d24284fedadd8e9e63cd27d05. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:38.535710 5126 raft_consensus.cc:1261] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:05:38.535962 5384 raft_consensus.cc:1261] T a726769d24284fedadd8e9e63cd27d05 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:05:38.536619 9009 consensus_queue.cc:1035] T a726769d24284fedadd8e9e63cd27d05 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:05:38.536996 9146 consensus_queue.cc:1035] T a726769d24284fedadd8e9e63cd27d05 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:05:38.541505 7287 raft_consensus.cc:1261] T de6c7753c8564107b858876fcfa5924d P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:05:38.542124 8277 consensus_queue.cc:1035] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:05:38.560688 6889 raft_consensus.cc:1261] T de6c7753c8564107b858876fcfa5924d P 938a24b0e6a040b7966225af26558ff8 [term 1 LEARNER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:05:38.561334 8277 consensus_queue.cc:1035] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:05:38.562645 5382 raft_consensus.cc:1261] T de6c7753c8564107b858876fcfa5924d P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:05:38.564127 8897 consensus_queue.cc:1035] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:05:38.578239 8897 raft_consensus.cc:2932] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:38.581357 8375 raft_consensus.cc:2932] T a726769d24284fedadd8e9e63cd27d05 P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } }
I20240501 14:05:38.582231 7285 raft_consensus.cc:2932] T de6c7753c8564107b858876fcfa5924d P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:38.585373 5125 raft_consensus.cc:2932] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } }
I20240501 14:05:38.587754 5382 raft_consensus.cc:2932] T a726769d24284fedadd8e9e63cd27d05 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } }
I20240501 14:05:38.588798 5385 raft_consensus.cc:2932] T de6c7753c8564107b858876fcfa5924d P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:38.594045 4853 catalog_manager.cc:5484] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 reported cstate change: config changed from index -1 to 4, NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) added. New cstate: current_term: 1 leader_uuid: "04088ee3bea440a7bd5ff772152b9410" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } }
I20240501 14:05:38.603896 4846 catalog_manager.cc:5484] T de6c7753c8564107b858876fcfa5924d P be56773105e744919601127a94e98adf reported cstate change: config changed from index 7 to 8, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:38.620271 6889 raft_consensus.cc:2932] T de6c7753c8564107b858876fcfa5924d P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:38.647361 5251 consensus_queue.cc:237] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:05:38.675379 6888 raft_consensus.cc:1261] T de6c7753c8564107b858876fcfa5924d P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:05:38.676661 9193 consensus_queue.cc:1035] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:05:38.680788 7287 raft_consensus.cc:1261] T de6c7753c8564107b858876fcfa5924d P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:05:38.681998 9193 consensus_queue.cc:1035] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:05:38.685225 8500 raft_consensus.cc:2932] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER be56773105e744919601127a94e98adf (127.29.124.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:38.689141 4838 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet de6c7753c8564107b858876fcfa5924d with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:05:38.689808 7287 raft_consensus.cc:2932] T de6c7753c8564107b858876fcfa5924d P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER be56773105e744919601127a94e98adf (127.29.124.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:38.689967 6888 raft_consensus.cc:2932] T de6c7753c8564107b858876fcfa5924d P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER be56773105e744919601127a94e98adf (127.29.124.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:38.695902 4854 catalog_manager.cc:5484] T de6c7753c8564107b858876fcfa5924d P 938a24b0e6a040b7966225af26558ff8 reported cstate change: config changed from index 8 to 9, VOTER be56773105e744919601127a94e98adf (127.29.124.132) evicted. New cstate: current_term: 1 leader_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:38.697839 5125 consensus_queue.cc:237] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } }
I20240501 14:05:38.701171 5365 tablet_service.cc:1514] Processing DeleteTablet for tablet de6c7753c8564107b858876fcfa5924d with delete_type TABLET_DATA_TOMBSTONED (TS be56773105e744919601127a94e98adf not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:43020
I20240501 14:05:38.706255 9211 tablet_replica.cc:330] T de6c7753c8564107b858876fcfa5924d P be56773105e744919601127a94e98adf: stopping tablet replica
I20240501 14:05:38.706547 9211 raft_consensus.cc:2229] T de6c7753c8564107b858876fcfa5924d P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:05:38.706888 9211 raft_consensus.cc:2258] T de6c7753c8564107b858876fcfa5924d P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Raft consensus is shut down!
W20240501 14:05:38.707505 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ac4b234f794149e4b6def7a3b358942d. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:38.708350 5385 raft_consensus.cc:1261] T ac4b234f794149e4b6def7a3b358942d P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:05:38.708545 9211 ts_tablet_manager.cc:1875] T de6c7753c8564107b858876fcfa5924d P be56773105e744919601127a94e98adf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:05:38.708926 9051 consensus_queue.cc:1035] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:05:38.716866 7155 raft_consensus.cc:1261] T ac4b234f794149e4b6def7a3b358942d P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:05:38.717527 9051 consensus_queue.cc:1035] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:05:38.719298 9211 ts_tablet_manager.cc:1888] T de6c7753c8564107b858876fcfa5924d P be56773105e744919601127a94e98adf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 14:05:38.720042 9211 log.cc:1196] T de6c7753c8564107b858876fcfa5924d P be56773105e744919601127a94e98adf: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/wal/wals/de6c7753c8564107b858876fcfa5924d
I20240501 14:05:38.720882 4839 catalog_manager.cc:4830] TS be56773105e744919601127a94e98adf (127.29.124.132:35459): tablet de6c7753c8564107b858876fcfa5924d (table rebalance_test_table_6 [id=3c1513e944ee4b518373c4efa5e200cc]) successfully deleted
I20240501 14:05:38.726006 5125 consensus_queue.cc:237] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } }
I20240501 14:05:38.731500 9048 raft_consensus.cc:2932] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:38.735628 5251 raft_consensus.cc:1261] T 3d6859d53fd747a5a5e94722626fa522 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:05:38.736733 9167 consensus_queue.cc:1035] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:05:38.739212 5384 raft_consensus.cc:2932] T ac4b234f794149e4b6def7a3b358942d P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:38.740159 6889 raft_consensus.cc:1261] T 3d6859d53fd747a5a5e94722626fa522 P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:05:38.742755 9048 consensus_queue.cc:1035] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 14:05:38.743623 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3d6859d53fd747a5a5e94722626fa522. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:38.748335 4854 catalog_manager.cc:5484] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 reported cstate change: config changed from index 6 to 7, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New cstate: current_term: 1 leader_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:05:38.752310 5255 consensus_queue.cc:237] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } }
I20240501 14:05:38.758746 9167 raft_consensus.cc:2932] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } }
I20240501 14:05:38.765030 5254 raft_consensus.cc:2932] T 3d6859d53fd747a5a5e94722626fa522 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } }
I20240501 14:05:38.771037 4846 catalog_manager.cc:5484] T 3d6859d53fd747a5a5e94722626fa522 P 1c3fdc5272484ac09ef3e5e36a08f27c reported cstate change: config changed from index 6 to 7, NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) added. New cstate: current_term: 1 leader_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } }
I20240501 14:05:38.776612 7156 raft_consensus.cc:2932] T ac4b234f794149e4b6def7a3b358942d P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:38.778990 6889 raft_consensus.cc:2932] T 3d6859d53fd747a5a5e94722626fa522 P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } }
I20240501 14:05:38.803524 5125 consensus_queue.cc:237] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } }
I20240501 14:05:38.836246 9228 ts_tablet_manager.cc:902] T 99d958617b7e4ef385b94f4c73f4dd84 P 4360b59e3a154688a79ca5f9c0a37bc9: Initiating tablet copy from peer 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129:44033)
I20240501 14:05:38.839551 9229 ts_tablet_manager.cc:902] T fee16905387d428aa02cd2070c86c790 P 19a6d2d8a08142d5bffcde3a37234692: Initiating tablet copy from peer 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129:44033)
I20240501 14:05:38.839833 7287 raft_consensus.cc:1261] T f9cb784f60774ea7b6e70ccfebf867a8 P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:05:38.840616 9167 consensus_queue.cc:1035] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:05:38.846544 9227 ts_tablet_manager.cc:902] T 420642af520d45ffbc7f2d35f93e75a7 P 938a24b0e6a040b7966225af26558ff8: Initiating tablet copy from peer be56773105e744919601127a94e98adf (127.29.124.132:35459)
W20240501 14:05:38.849143 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f9cb784f60774ea7b6e70ccfebf867a8. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:38.851884 5384 raft_consensus.cc:1261] T f9cb784f60774ea7b6e70ccfebf867a8 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:05:38.852631 9167 consensus_queue.cc:1035] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:05:38.855760 9051 raft_consensus.cc:2932] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } }
I20240501 14:05:38.856249 5384 raft_consensus.cc:2932] T f9cb784f60774ea7b6e70ccfebf867a8 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } }
I20240501 14:05:38.861989 4847 catalog_manager.cc:5484] T f9cb784f60774ea7b6e70ccfebf867a8 P be56773105e744919601127a94e98adf reported cstate change: config changed from index 6 to 7, NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) added. New cstate: current_term: 1 leader_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } }
I20240501 14:05:38.864758 7287 raft_consensus.cc:2932] T f9cb784f60774ea7b6e70ccfebf867a8 P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } }
I20240501 14:05:38.872344 5384 raft_consensus.cc:1261] T 1a7bbfc15a76450bbc9197037e856e08 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:05:38.873179 8897 consensus_queue.cc:1035] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 14:05:38.874716 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1a7bbfc15a76450bbc9197037e856e08. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:38.875303 7025 raft_consensus.cc:1261] T 1a7bbfc15a76450bbc9197037e856e08 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:05:38.875860 8897 consensus_queue.cc:1035] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:05:38.889182 8897 raft_consensus.cc:2932] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:38.894223 5385 raft_consensus.cc:2932] T 1a7bbfc15a76450bbc9197037e856e08 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:38.901008 4846 catalog_manager.cc:5484] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c reported cstate change: config changed from index 6 to 7, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) added. New cstate: current_term: 1 leader_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:05:38.910745 7025 raft_consensus.cc:2932] T 1a7bbfc15a76450bbc9197037e856e08 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:38.924194 9228 tablet_copy_client.cc:308] T 99d958617b7e4ef385b94f4c73f4dd84 P 4360b59e3a154688a79ca5f9c0a37bc9: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.129:44033
I20240501 14:05:38.924988 5015 tablet_copy_service.cc:140] P 04088ee3bea440a7bd5ff772152b9410: Received BeginTabletCopySession request for tablet 99d958617b7e4ef385b94f4c73f4dd84 from peer 4360b59e3a154688a79ca5f9c0a37bc9 ({username='slave'} at 127.29.124.135:54069)
I20240501 14:05:38.925176 5015 tablet_copy_service.cc:161] P 04088ee3bea440a7bd5ff772152b9410: Beginning new tablet copy session on tablet 99d958617b7e4ef385b94f4c73f4dd84 from peer 4360b59e3a154688a79ca5f9c0a37bc9 at {username='slave'} at 127.29.124.135:54069: session id = 4360b59e3a154688a79ca5f9c0a37bc9-99d958617b7e4ef385b94f4c73f4dd84
I20240501 14:05:38.928618 5015 tablet_copy_source_session.cc:205] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:05:38.930594 9227 tablet_copy_client.cc:308] T 420642af520d45ffbc7f2d35f93e75a7 P 938a24b0e6a040b7966225af26558ff8: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.132:35459
I20240501 14:05:38.931461 5406 tablet_copy_service.cc:140] P be56773105e744919601127a94e98adf: Received BeginTabletCopySession request for tablet 420642af520d45ffbc7f2d35f93e75a7 from peer 938a24b0e6a040b7966225af26558ff8 ({username='slave'} at 127.29.124.133:50583)
I20240501 14:05:38.931645 5406 tablet_copy_service.cc:161] P be56773105e744919601127a94e98adf: Beginning new tablet copy session on tablet 420642af520d45ffbc7f2d35f93e75a7 from peer 938a24b0e6a040b7966225af26558ff8 at {username='slave'} at 127.29.124.133:50583: session id = 938a24b0e6a040b7966225af26558ff8-420642af520d45ffbc7f2d35f93e75a7
I20240501 14:05:38.934504 5406 tablet_copy_source_session.cc:205] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:05:38.938059 9229 tablet_copy_client.cc:308] T fee16905387d428aa02cd2070c86c790 P 19a6d2d8a08142d5bffcde3a37234692: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.129:44033
I20240501 14:05:38.938928 5015 tablet_copy_service.cc:140] P 04088ee3bea440a7bd5ff772152b9410: Received BeginTabletCopySession request for tablet fee16905387d428aa02cd2070c86c790 from peer 19a6d2d8a08142d5bffcde3a37234692 ({username='slave'} at 127.29.124.134:36629)
I20240501 14:05:38.939188 5015 tablet_copy_service.cc:161] P 04088ee3bea440a7bd5ff772152b9410: Beginning new tablet copy session on tablet fee16905387d428aa02cd2070c86c790 from peer 19a6d2d8a08142d5bffcde3a37234692 at {username='slave'} at 127.29.124.134:36629: session id = 19a6d2d8a08142d5bffcde3a37234692-fee16905387d428aa02cd2070c86c790
I20240501 14:05:38.942494 5125 consensus_queue.cc:237] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } }
I20240501 14:05:38.942736 5015 tablet_copy_source_session.cc:205] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:05:38.944648 9228 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99d958617b7e4ef385b94f4c73f4dd84. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 14:05:38.946197 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 88ed5438fc934adc9fb98c6cb4476598. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:38.947247 4993 raft_consensus.cc:1261] T 88ed5438fc934adc9fb98c6cb4476598 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:05:38.947510 5255 raft_consensus.cc:1261] T 88ed5438fc934adc9fb98c6cb4476598 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:05:38.947850 9057 consensus_queue.cc:1035] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:05:38.948313 9051 consensus_queue.cc:1035] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:05:38.952250 9227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 420642af520d45ffbc7f2d35f93e75a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:38.955338 9229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fee16905387d428aa02cd2070c86c790. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:38.959787 9255 ts_tablet_manager.cc:902] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 6dac90f01e744f9c8701a404440a4c77: Initiating tablet copy from peer 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209)
I20240501 14:05:38.963574 9229 tablet_copy_client.cc:783] T fee16905387d428aa02cd2070c86c790 P 19a6d2d8a08142d5bffcde3a37234692: tablet copy: Starting download of 0 data blocks...
I20240501 14:05:38.963840 9229 tablet_copy_client.cc:647] T fee16905387d428aa02cd2070c86c790 P 19a6d2d8a08142d5bffcde3a37234692: tablet copy: Starting download of 1 WAL segments...
I20240501 14:05:38.975396 9272 ts_tablet_manager.cc:902] T 43b46fcf827c4275a47291afb107c290 P 6dac90f01e744f9c8701a404440a4c77: Initiating tablet copy from peer e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377)
I20240501 14:05:38.976101 9051 raft_consensus.cc:2932] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:38.982687 5254 raft_consensus.cc:2932] T 88ed5438fc934adc9fb98c6cb4476598 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:38.983569 9228 tablet_copy_client.cc:783] T 99d958617b7e4ef385b94f4c73f4dd84 P 4360b59e3a154688a79ca5f9c0a37bc9: tablet copy: Starting download of 0 data blocks...
I20240501 14:05:38.983805 9228 tablet_copy_client.cc:647] T 99d958617b7e4ef385b94f4c73f4dd84 P 4360b59e3a154688a79ca5f9c0a37bc9: tablet copy: Starting download of 1 WAL segments...
I20240501 14:05:38.983909 4854 catalog_manager.cc:5484] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 reported cstate change: config changed from index -1 to 4, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) added. New cstate: current_term: 1 leader_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:05:38.989333 9229 tablet_copy_client.cc:523] T fee16905387d428aa02cd2070c86c790 P 19a6d2d8a08142d5bffcde3a37234692: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:05:38.992748 4992 consensus_queue.cc:237] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }
I20240501 14:05:38.996845 9229 tablet_bootstrap.cc:492] T fee16905387d428aa02cd2070c86c790 P 19a6d2d8a08142d5bffcde3a37234692: Bootstrap starting.
I20240501 14:05:39.002871 9270 ts_tablet_manager.cc:902] T c8a961e711114f369ba06840c062b982 P 4360b59e3a154688a79ca5f9c0a37bc9: Initiating tablet copy from peer e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377)
I20240501 14:05:39.007983 4994 raft_consensus.cc:2932] T 88ed5438fc934adc9fb98c6cb4476598 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:39.016651 5254 raft_consensus.cc:1261] T 63650547ab444f45bb377bcb49f258ad P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:05:39.016956 5125 raft_consensus.cc:1261] T 63650547ab444f45bb377bcb49f258ad P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:05:39.018146 9146 consensus_queue.cc:1035] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:05:39.019265 9146 consensus_queue.cc:1035] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240501 14:05:39.021735 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63650547ab444f45bb377bcb49f258ad. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:39.022459 9227 tablet_copy_client.cc:783] T 420642af520d45ffbc7f2d35f93e75a7 P 938a24b0e6a040b7966225af26558ff8: tablet copy: Starting download of 0 data blocks...
I20240501 14:05:39.022822 9227 tablet_copy_client.cc:647] T 420642af520d45ffbc7f2d35f93e75a7 P 938a24b0e6a040b7966225af26558ff8: tablet copy: Starting download of 1 WAL segments...
I20240501 14:05:39.024828 9228 tablet_copy_client.cc:523] T 99d958617b7e4ef385b94f4c73f4dd84 P 4360b59e3a154688a79ca5f9c0a37bc9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:05:39.028360 9227 tablet_copy_client.cc:523] T 420642af520d45ffbc7f2d35f93e75a7 P 938a24b0e6a040b7966225af26558ff8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:05:39.033404 9227 tablet_bootstrap.cc:492] T 420642af520d45ffbc7f2d35f93e75a7 P 938a24b0e6a040b7966225af26558ff8: Bootstrap starting.
I20240501 14:05:39.039425 9281 ts_tablet_manager.cc:902] T f4d2b8c005b64bbb8e6621da068d15e5 P 4360b59e3a154688a79ca5f9c0a37bc9: Initiating tablet copy from peer be56773105e744919601127a94e98adf (127.29.124.132:35459)
I20240501 14:05:39.042622 9146 raft_consensus.cc:2932] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:39.047420 5126 raft_consensus.cc:2932] T 63650547ab444f45bb377bcb49f258ad P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:39.050652 5254 raft_consensus.cc:2932] T 63650547ab444f45bb377bcb49f258ad P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:39.057061 4846 catalog_manager.cc:5484] T 63650547ab444f45bb377bcb49f258ad P e9c60449cf3c46f8aca4ab6c535302a4 reported cstate change: config changed from index -1 to 4, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New cstate: current_term: 1 leader_uuid: "04088ee3bea440a7bd5ff772152b9410" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:39.060112 9296 ts_tablet_manager.cc:902] T c8a961e711114f369ba06840c062b982 P 938a24b0e6a040b7966225af26558ff8: Initiating tablet copy from peer e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377)
I20240501 14:05:39.060662 9255 tablet_copy_client.cc:308] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 6dac90f01e744f9c8701a404440a4c77: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.131:33209
I20240501 14:05:39.061697 5276 tablet_copy_service.cc:140] P 1c3fdc5272484ac09ef3e5e36a08f27c: Received BeginTabletCopySession request for tablet 23ca2bf1e59f44fcac8a48aff8d056b1 from peer 6dac90f01e744f9c8701a404440a4c77 ({username='slave'} at 127.29.124.136:49141)
I20240501 14:05:39.067019 5276 tablet_copy_service.cc:161] P 1c3fdc5272484ac09ef3e5e36a08f27c: Beginning new tablet copy session on tablet 23ca2bf1e59f44fcac8a48aff8d056b1 from peer 6dac90f01e744f9c8701a404440a4c77 at {username='slave'} at 127.29.124.136:49141: session id = 6dac90f01e744f9c8701a404440a4c77-23ca2bf1e59f44fcac8a48aff8d056b1
I20240501 14:05:39.070328 5276 tablet_copy_source_session.cc:205] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:05:39.071808 9228 tablet_bootstrap.cc:492] T 99d958617b7e4ef385b94f4c73f4dd84 P 4360b59e3a154688a79ca5f9c0a37bc9: Bootstrap starting.
I20240501 14:05:39.072567 9255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23ca2bf1e59f44fcac8a48aff8d056b1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:39.090313 9296 tablet_copy_client.cc:308] T c8a961e711114f369ba06840c062b982 P 938a24b0e6a040b7966225af26558ff8: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.130:34377
I20240501 14:05:39.091073 9255 tablet_copy_client.cc:783] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 6dac90f01e744f9c8701a404440a4c77: tablet copy: Starting download of 0 data blocks...
I20240501 14:05:39.091372 5146 tablet_copy_service.cc:140] P e9c60449cf3c46f8aca4ab6c535302a4: Received BeginTabletCopySession request for tablet c8a961e711114f369ba06840c062b982 from peer 938a24b0e6a040b7966225af26558ff8 ({username='slave'} at 127.29.124.133:48619)
I20240501 14:05:39.091547 5146 tablet_copy_service.cc:161] P e9c60449cf3c46f8aca4ab6c535302a4: Beginning new tablet copy session on tablet c8a961e711114f369ba06840c062b982 from peer 938a24b0e6a040b7966225af26558ff8 at {username='slave'} at 127.29.124.133:48619: session id = 938a24b0e6a040b7966225af26558ff8-c8a961e711114f369ba06840c062b982
I20240501 14:05:39.094641 5146 tablet_copy_source_session.cc:205] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:05:39.096073 9296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8a961e711114f369ba06840c062b982. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:39.098908 9304 ts_tablet_manager.cc:902] T a726769d24284fedadd8e9e63cd27d05 P 19a6d2d8a08142d5bffcde3a37234692: Initiating tablet copy from peer 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129:44033)
I20240501 14:05:39.103027 9255 tablet_copy_client.cc:647] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 6dac90f01e744f9c8701a404440a4c77: tablet copy: Starting download of 1 WAL segments...
I20240501 14:05:39.100616 9272 tablet_copy_client.cc:308] T 43b46fcf827c4275a47291afb107c290 P 6dac90f01e744f9c8701a404440a4c77: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.130:34377
I20240501 14:05:39.107159 5146 tablet_copy_service.cc:140] P e9c60449cf3c46f8aca4ab6c535302a4: Received BeginTabletCopySession request for tablet 43b46fcf827c4275a47291afb107c290 from peer 6dac90f01e744f9c8701a404440a4c77 ({username='slave'} at 127.29.124.136:51401)
I20240501 14:05:39.107343 5146 tablet_copy_service.cc:161] P e9c60449cf3c46f8aca4ab6c535302a4: Beginning new tablet copy session on tablet 43b46fcf827c4275a47291afb107c290 from peer 6dac90f01e744f9c8701a404440a4c77 at {username='slave'} at 127.29.124.136:51401: session id = 6dac90f01e744f9c8701a404440a4c77-43b46fcf827c4275a47291afb107c290
I20240501 14:05:39.109697 9255 tablet_copy_client.cc:523] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 6dac90f01e744f9c8701a404440a4c77: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:05:39.110422 5146 tablet_copy_source_session.cc:205] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:05:39.111672 9272 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 43b46fcf827c4275a47291afb107c290. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:39.113129 9296 tablet_copy_client.cc:783] T c8a961e711114f369ba06840c062b982 P 938a24b0e6a040b7966225af26558ff8: tablet copy: Starting download of 0 data blocks...
I20240501 14:05:39.115607 9296 tablet_copy_client.cc:647] T c8a961e711114f369ba06840c062b982 P 938a24b0e6a040b7966225af26558ff8: tablet copy: Starting download of 1 WAL segments...
I20240501 14:05:39.117405 9270 tablet_copy_client.cc:308] T c8a961e711114f369ba06840c062b982 P 4360b59e3a154688a79ca5f9c0a37bc9: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.130:34377
I20240501 14:05:39.117753 9296 tablet_copy_client.cc:523] T c8a961e711114f369ba06840c062b982 P 938a24b0e6a040b7966225af26558ff8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:05:39.118243 5146 tablet_copy_service.cc:140] P e9c60449cf3c46f8aca4ab6c535302a4: Received BeginTabletCopySession request for tablet c8a961e711114f369ba06840c062b982 from peer 4360b59e3a154688a79ca5f9c0a37bc9 ({username='slave'} at 127.29.124.135:58119)
I20240501 14:05:39.118393 5146 tablet_copy_service.cc:161] P e9c60449cf3c46f8aca4ab6c535302a4: Beginning new tablet copy session on tablet c8a961e711114f369ba06840c062b982 from peer 4360b59e3a154688a79ca5f9c0a37bc9 at {username='slave'} at 127.29.124.135:58119: session id = 4360b59e3a154688a79ca5f9c0a37bc9-c8a961e711114f369ba06840c062b982
I20240501 14:05:39.121415 5146 tablet_copy_source_session.cc:205] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:05:39.123880 9270 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8a961e711114f369ba06840c062b982. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:39.145192 9255 tablet_bootstrap.cc:492] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 6dac90f01e744f9c8701a404440a4c77: Bootstrap starting.
I20240501 14:05:39.151322 4973 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:39.153950 9296 tablet_bootstrap.cc:492] T c8a961e711114f369ba06840c062b982 P 938a24b0e6a040b7966225af26558ff8: Bootstrap starting.
I20240501 14:05:39.159821 9272 tablet_copy_client.cc:783] T 43b46fcf827c4275a47291afb107c290 P 6dac90f01e744f9c8701a404440a4c77: tablet copy: Starting download of 0 data blocks...
I20240501 14:05:39.161702 9227 tablet_bootstrap.cc:492] T 420642af520d45ffbc7f2d35f93e75a7 P 938a24b0e6a040b7966225af26558ff8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:05:39.164314 5236 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:39.181916 9227 tablet_bootstrap.cc:492] T 420642af520d45ffbc7f2d35f93e75a7 P 938a24b0e6a040b7966225af26558ff8: Bootstrap complete.
I20240501 14:05:39.182482 9227 ts_tablet_manager.cc:1367] T 420642af520d45ffbc7f2d35f93e75a7 P 938a24b0e6a040b7966225af26558ff8: Time spent bootstrapping tablet: real 0.149s user 0.047s sys 0.002s
I20240501 14:05:39.183557 9227 raft_consensus.cc:348] T 420642af520d45ffbc7f2d35f93e75a7 P 938a24b0e6a040b7966225af26558ff8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }
I20240501 14:05:39.185845 9227 raft_consensus.cc:729] T 420642af520d45ffbc7f2d35f93e75a7 P 938a24b0e6a040b7966225af26558ff8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 938a24b0e6a040b7966225af26558ff8, State: Initialized, Role: LEARNER
I20240501 14:05:39.187204 9227 consensus_queue.cc:260] T 420642af520d45ffbc7f2d35f93e75a7 P 938a24b0e6a040b7966225af26558ff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }
I20240501 14:05:39.188074 9227 ts_tablet_manager.cc:1398] T 420642af520d45ffbc7f2d35f93e75a7 P 938a24b0e6a040b7966225af26558ff8: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240501 14:05:39.189162 9272 tablet_copy_client.cc:647] T 43b46fcf827c4275a47291afb107c290 P 6dac90f01e744f9c8701a404440a4c77: tablet copy: Starting download of 1 WAL segments...
I20240501 14:05:39.189746 9270 tablet_copy_client.cc:783] T c8a961e711114f369ba06840c062b982 P 4360b59e3a154688a79ca5f9c0a37bc9: tablet copy: Starting download of 0 data blocks...
I20240501 14:05:39.190078 9270 tablet_copy_client.cc:647] T c8a961e711114f369ba06840c062b982 P 4360b59e3a154688a79ca5f9c0a37bc9: tablet copy: Starting download of 1 WAL segments...
I20240501 14:05:39.192821 9304 tablet_copy_client.cc:308] T a726769d24284fedadd8e9e63cd27d05 P 19a6d2d8a08142d5bffcde3a37234692: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.129:44033
I20240501 14:05:39.192971 5406 tablet_copy_service.cc:342] P be56773105e744919601127a94e98adf: Request end of tablet copy session 938a24b0e6a040b7966225af26558ff8-420642af520d45ffbc7f2d35f93e75a7 received from {username='slave'} at 127.29.124.133:50583
I20240501 14:05:39.193125 5406 tablet_copy_service.cc:433] P be56773105e744919601127a94e98adf: ending tablet copy session 938a24b0e6a040b7966225af26558ff8-420642af520d45ffbc7f2d35f93e75a7 on tablet 420642af520d45ffbc7f2d35f93e75a7 with peer 938a24b0e6a040b7966225af26558ff8
I20240501 14:05:39.193708 9272 tablet_copy_client.cc:523] T 43b46fcf827c4275a47291afb107c290 P 6dac90f01e744f9c8701a404440a4c77: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:05:39.195046 9270 tablet_copy_client.cc:523] T c8a961e711114f369ba06840c062b982 P 4360b59e3a154688a79ca5f9c0a37bc9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:05:39.197539 5015 tablet_copy_service.cc:140] P 04088ee3bea440a7bd5ff772152b9410: Received BeginTabletCopySession request for tablet a726769d24284fedadd8e9e63cd27d05 from peer 19a6d2d8a08142d5bffcde3a37234692 ({username='slave'} at 127.29.124.134:36629)
I20240501 14:05:39.197748 5015 tablet_copy_service.cc:161] P 04088ee3bea440a7bd5ff772152b9410: Beginning new tablet copy session on tablet a726769d24284fedadd8e9e63cd27d05 from peer 19a6d2d8a08142d5bffcde3a37234692 at {username='slave'} at 127.29.124.134:36629: session id = 19a6d2d8a08142d5bffcde3a37234692-a726769d24284fedadd8e9e63cd27d05
I20240501 14:05:39.200760 5015 tablet_copy_source_session.cc:205] T a726769d24284fedadd8e9e63cd27d05 P 04088ee3bea440a7bd5ff772152b9410: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:05:39.202443 9304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a726769d24284fedadd8e9e63cd27d05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:39.202893 9281 tablet_copy_client.cc:308] T f4d2b8c005b64bbb8e6621da068d15e5 P 4360b59e3a154688a79ca5f9c0a37bc9: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.132:35459
I20240501 14:05:39.203749 5406 tablet_copy_service.cc:140] P be56773105e744919601127a94e98adf: Received BeginTabletCopySession request for tablet f4d2b8c005b64bbb8e6621da068d15e5 from peer 4360b59e3a154688a79ca5f9c0a37bc9 ({username='slave'} at 127.29.124.135:51309)
I20240501 14:05:39.203948 5406 tablet_copy_service.cc:161] P be56773105e744919601127a94e98adf: Beginning new tablet copy session on tablet f4d2b8c005b64bbb8e6621da068d15e5 from peer 4360b59e3a154688a79ca5f9c0a37bc9 at {username='slave'} at 127.29.124.135:51309: session id = 4360b59e3a154688a79ca5f9c0a37bc9-f4d2b8c005b64bbb8e6621da068d15e5
I20240501 14:05:39.207218 5406 tablet_copy_source_session.cc:205] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:05:39.210134 9334 ts_tablet_manager.cc:902] T ac4b234f794149e4b6def7a3b358942d P 6dac90f01e744f9c8701a404440a4c77: Initiating tablet copy from peer e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377)
I20240501 14:05:39.214318 9270 tablet_bootstrap.cc:492] T c8a961e711114f369ba06840c062b982 P 4360b59e3a154688a79ca5f9c0a37bc9: Bootstrap starting.
I20240501 14:05:39.219125 9334 tablet_copy_client.cc:308] T ac4b234f794149e4b6def7a3b358942d P 6dac90f01e744f9c8701a404440a4c77: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.130:34377
I20240501 14:05:39.220480 5146 tablet_copy_service.cc:140] P e9c60449cf3c46f8aca4ab6c535302a4: Received BeginTabletCopySession request for tablet ac4b234f794149e4b6def7a3b358942d from peer 6dac90f01e744f9c8701a404440a4c77 ({username='slave'} at 127.29.124.136:51401)
I20240501 14:05:39.220657 5146 tablet_copy_service.cc:161] P e9c60449cf3c46f8aca4ab6c535302a4: Beginning new tablet copy session on tablet ac4b234f794149e4b6def7a3b358942d from peer 6dac90f01e744f9c8701a404440a4c77 at {username='slave'} at 127.29.124.136:51401: session id = 6dac90f01e744f9c8701a404440a4c77-ac4b234f794149e4b6def7a3b358942d
I20240501 14:05:39.223452 5146 tablet_copy_source_session.cc:205] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:05:39.224936 9272 tablet_bootstrap.cc:492] T 43b46fcf827c4275a47291afb107c290 P 6dac90f01e744f9c8701a404440a4c77: Bootstrap starting.
I20240501 14:05:39.226019 9334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac4b234f794149e4b6def7a3b358942d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:39.214900 9281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4d2b8c005b64bbb8e6621da068d15e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:39.235834 9333 ts_tablet_manager.cc:902] T 3d6859d53fd747a5a5e94722626fa522 P 19a6d2d8a08142d5bffcde3a37234692: Initiating tablet copy from peer e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377)
I20240501 14:05:39.242707 9229 tablet_bootstrap.cc:492] T fee16905387d428aa02cd2070c86c790 P 19a6d2d8a08142d5bffcde3a37234692: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:05:39.243185 9229 tablet_bootstrap.cc:492] T fee16905387d428aa02cd2070c86c790 P 19a6d2d8a08142d5bffcde3a37234692: Bootstrap complete.
I20240501 14:05:39.243479 9229 ts_tablet_manager.cc:1367] T fee16905387d428aa02cd2070c86c790 P 19a6d2d8a08142d5bffcde3a37234692: Time spent bootstrapping tablet: real 0.247s user 0.073s sys 0.028s
I20240501 14:05:39.244249 9229 raft_consensus.cc:348] T fee16905387d428aa02cd2070c86c790 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } }
I20240501 14:05:39.244540 9229 raft_consensus.cc:729] T fee16905387d428aa02cd2070c86c790 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 19a6d2d8a08142d5bffcde3a37234692, State: Initialized, Role: LEARNER
I20240501 14:05:39.245272 9229 consensus_queue.cc:260] T fee16905387d428aa02cd2070c86c790 P 19a6d2d8a08142d5bffcde3a37234692 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 1.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } }
I20240501 14:05:39.246814 9229 ts_tablet_manager.cc:1398] T fee16905387d428aa02cd2070c86c790 P 19a6d2d8a08142d5bffcde3a37234692: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240501 14:05:39.249723 5015 tablet_copy_service.cc:342] P 04088ee3bea440a7bd5ff772152b9410: Request end of tablet copy session 19a6d2d8a08142d5bffcde3a37234692-fee16905387d428aa02cd2070c86c790 received from {username='slave'} at 127.29.124.134:36629
I20240501 14:05:39.249864 5015 tablet_copy_service.cc:433] P 04088ee3bea440a7bd5ff772152b9410: ending tablet copy session 19a6d2d8a08142d5bffcde3a37234692-fee16905387d428aa02cd2070c86c790 on tablet fee16905387d428aa02cd2070c86c790 with peer 19a6d2d8a08142d5bffcde3a37234692
I20240501 14:05:39.264166 9304 tablet_copy_client.cc:783] T a726769d24284fedadd8e9e63cd27d05 P 19a6d2d8a08142d5bffcde3a37234692: tablet copy: Starting download of 0 data blocks...
I20240501 14:05:39.272490 5365 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:39.274058 5385 consensus_queue.cc:237] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }
I20240501 14:05:39.280679 6872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:39.280836 5106 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:39.267604 7004 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:39.301643 9304 tablet_copy_client.cc:647] T a726769d24284fedadd8e9e63cd27d05 P 19a6d2d8a08142d5bffcde3a37234692: tablet copy: Starting download of 1 WAL segments...
I20240501 14:05:39.325222 9304 tablet_copy_client.cc:523] T a726769d24284fedadd8e9e63cd27d05 P 19a6d2d8a08142d5bffcde3a37234692: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:05:39.327764 9334 tablet_copy_client.cc:783] T ac4b234f794149e4b6def7a3b358942d P 6dac90f01e744f9c8701a404440a4c77: tablet copy: Starting download of 0 data blocks...
I20240501 14:05:39.328049 9334 tablet_copy_client.cc:647] T ac4b234f794149e4b6def7a3b358942d P 6dac90f01e744f9c8701a404440a4c77: tablet copy: Starting download of 1 WAL segments...
I20240501 14:05:39.329582 5382 consensus_queue.cc:237] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } }
I20240501 14:05:39.331362 9334 tablet_copy_client.cc:523] T ac4b234f794149e4b6def7a3b358942d P 6dac90f01e744f9c8701a404440a4c77: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:05:39.335675 9334 tablet_bootstrap.cc:492] T ac4b234f794149e4b6def7a3b358942d P 6dac90f01e744f9c8701a404440a4c77: Bootstrap starting.
I20240501 14:05:39.350337 6889 raft_consensus.cc:1203] T 420642af520d45ffbc7f2d35f93e75a7 P 938a24b0e6a040b7966225af26558ff8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:05:39.367092 9304 tablet_bootstrap.cc:492] T a726769d24284fedadd8e9e63cd27d05 P 19a6d2d8a08142d5bffcde3a37234692: Bootstrap starting.
I20240501 14:05:39.372264 9255 tablet_bootstrap.cc:492] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 6dac90f01e744f9c8701a404440a4c77: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:05:39.372718 9255 tablet_bootstrap.cc:492] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 6dac90f01e744f9c8701a404440a4c77: Bootstrap complete.
I20240501 14:05:39.373005 9255 ts_tablet_manager.cc:1367] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 6dac90f01e744f9c8701a404440a4c77: Time spent bootstrapping tablet: real 0.228s user 0.050s sys 0.000s
I20240501 14:05:39.373907 9255 raft_consensus.cc:348] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } }
I20240501 14:05:39.374192 9255 raft_consensus.cc:729] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6dac90f01e744f9c8701a404440a4c77, State: Initialized, Role: LEARNER
I20240501 14:05:39.374634 9255 consensus_queue.cc:260] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 6dac90f01e744f9c8701a404440a4c77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } }
I20240501 14:05:39.381691 9255 ts_tablet_manager.cc:1398] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 6dac90f01e744f9c8701a404440a4c77: Time spent starting tablet: real 0.009s user 0.006s sys 0.003s
I20240501 14:05:39.383329 5276 tablet_copy_service.cc:342] P 1c3fdc5272484ac09ef3e5e36a08f27c: Request end of tablet copy session 6dac90f01e744f9c8701a404440a4c77-23ca2bf1e59f44fcac8a48aff8d056b1 received from {username='slave'} at 127.29.124.136:49141
I20240501 14:05:39.383477 5276 tablet_copy_service.cc:433] P 1c3fdc5272484ac09ef3e5e36a08f27c: ending tablet copy session 6dac90f01e744f9c8701a404440a4c77-23ca2bf1e59f44fcac8a48aff8d056b1 on tablet 23ca2bf1e59f44fcac8a48aff8d056b1 with peer 6dac90f01e744f9c8701a404440a4c77
W20240501 14:05:39.394043 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b284aa82594a44f5885f32fa05844913. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:39.394044 4993 raft_consensus.cc:1261] T b284aa82594a44f5885f32fa05844913 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:05:39.394834 8891 consensus_queue.cc:1035] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:05:39.424924 9227 ts_tablet_manager.cc:902] T 63650547ab444f45bb377bcb49f258ad P 938a24b0e6a040b7966225af26558ff8: Initiating tablet copy from peer 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129:44033)
I20240501 14:05:39.428190 9381 ts_tablet_manager.cc:902] T 1a7bbfc15a76450bbc9197037e856e08 P 4360b59e3a154688a79ca5f9c0a37bc9: Initiating tablet copy from peer 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209)
W20240501 14:05:39.428229 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9b88c43cb8704583a3e65c5c35c205f6. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:39.431002 5254 raft_consensus.cc:1261] T 9b88c43cb8704583a3e65c5c35c205f6 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:05:39.431319 6888 raft_consensus.cc:1261] T 9b88c43cb8704583a3e65c5c35c205f6 P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:05:39.431638 8891 consensus_queue.cc:1035] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:05:39.432588 8891 consensus_queue.cc:1035] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:05:39.443823 9296 tablet_bootstrap.cc:492] T c8a961e711114f369ba06840c062b982 P 938a24b0e6a040b7966225af26558ff8: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:05:39.444272 9296 tablet_bootstrap.cc:492] T c8a961e711114f369ba06840c062b982 P 938a24b0e6a040b7966225af26558ff8: Bootstrap complete.
I20240501 14:05:39.444582 9296 ts_tablet_manager.cc:1367] T c8a961e711114f369ba06840c062b982 P 938a24b0e6a040b7966225af26558ff8: Time spent bootstrapping tablet: real 0.291s user 0.044s sys 0.007s
I20240501 14:05:39.445441 9296 raft_consensus.cc:348] T c8a961e711114f369ba06840c062b982 P 938a24b0e6a040b7966225af26558ff8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }
I20240501 14:05:39.445760 9296 raft_consensus.cc:729] T c8a961e711114f369ba06840c062b982 P 938a24b0e6a040b7966225af26558ff8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 938a24b0e6a040b7966225af26558ff8, State: Initialized, Role: LEARNER
I20240501 14:05:39.445956 9296 consensus_queue.cc:260] T c8a961e711114f369ba06840c062b982 P 938a24b0e6a040b7966225af26558ff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }
I20240501 14:05:39.445863 8804 raft_consensus.cc:2932] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:39.446524 9296 ts_tablet_manager.cc:1398] T c8a961e711114f369ba06840c062b982 P 938a24b0e6a040b7966225af26558ff8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:05:39.447299 5146 tablet_copy_service.cc:342] P e9c60449cf3c46f8aca4ab6c535302a4: Request end of tablet copy session 938a24b0e6a040b7966225af26558ff8-c8a961e711114f369ba06840c062b982 received from {username='slave'} at 127.29.124.133:48619
I20240501 14:05:39.447415 5146 tablet_copy_service.cc:433] P e9c60449cf3c46f8aca4ab6c535302a4: ending tablet copy session 938a24b0e6a040b7966225af26558ff8-c8a961e711114f369ba06840c062b982 on tablet c8a961e711114f369ba06840c062b982 with peer 938a24b0e6a040b7966225af26558ff8
I20240501 14:05:39.452111 5125 raft_consensus.cc:1261] T b284aa82594a44f5885f32fa05844913 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240501 14:05:39.454893 8891 consensus_queue.cc:1035] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240501 14:05:39.455184 7136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:39.455178 7267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:39.471637 4993 raft_consensus.cc:2932] T b284aa82594a44f5885f32fa05844913 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:39.476650 9272 tablet_bootstrap.cc:492] T 43b46fcf827c4275a47291afb107c290 P 6dac90f01e744f9c8701a404440a4c77: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:05:39.477131 9272 tablet_bootstrap.cc:492] T 43b46fcf827c4275a47291afb107c290 P 6dac90f01e744f9c8701a404440a4c77: Bootstrap complete.
I20240501 14:05:39.477458 9272 ts_tablet_manager.cc:1367] T 43b46fcf827c4275a47291afb107c290 P 6dac90f01e744f9c8701a404440a4c77: Time spent bootstrapping tablet: real 0.253s user 0.052s sys 0.000s
I20240501 14:05:39.478328 9272 raft_consensus.cc:348] T 43b46fcf827c4275a47291afb107c290 P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } }
I20240501 14:05:39.478643 9272 raft_consensus.cc:729] T 43b46fcf827c4275a47291afb107c290 P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6dac90f01e744f9c8701a404440a4c77, State: Initialized, Role: LEARNER
I20240501 14:05:39.484174 4854 catalog_manager.cc:5484] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf reported cstate change: config changed from index -1 to 4, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:05:39.485436 9272 consensus_queue.cc:260] T 43b46fcf827c4275a47291afb107c290 P 6dac90f01e744f9c8701a404440a4c77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } }
I20240501 14:05:39.486099 9272 ts_tablet_manager.cc:1398] T 43b46fcf827c4275a47291afb107c290 P 6dac90f01e744f9c8701a404440a4c77: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20240501 14:05:39.486773 5146 tablet_copy_service.cc:342] P e9c60449cf3c46f8aca4ab6c535302a4: Request end of tablet copy session 6dac90f01e744f9c8701a404440a4c77-43b46fcf827c4275a47291afb107c290 received from {username='slave'} at 127.29.124.136:51401
I20240501 14:05:39.486898 5146 tablet_copy_service.cc:433] P e9c60449cf3c46f8aca4ab6c535302a4: ending tablet copy session 6dac90f01e744f9c8701a404440a4c77-43b46fcf827c4275a47291afb107c290 on tablet 43b46fcf827c4275a47291afb107c290 with peer 6dac90f01e744f9c8701a404440a4c77
I20240501 14:05:39.442889 9229 ts_tablet_manager.cc:902] T f9cb784f60774ea7b6e70ccfebf867a8 P 19a6d2d8a08142d5bffcde3a37234692: Initiating tablet copy from peer e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377)
I20240501 14:05:39.540181 8804 raft_consensus.cc:2932] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:39.540897 5255 raft_consensus.cc:2932] T 9b88c43cb8704583a3e65c5c35c205f6 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:39.546887 4854 catalog_manager.cc:5484] T 9b88c43cb8704583a3e65c5c35c205f6 P 1c3fdc5272484ac09ef3e5e36a08f27c reported cstate change: config changed from index 8 to 9, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:39.562930 9281 tablet_copy_client.cc:783] T f4d2b8c005b64bbb8e6621da068d15e5 P 4360b59e3a154688a79ca5f9c0a37bc9: tablet copy: Starting download of 0 data blocks...
I20240501 14:05:39.563259 9281 tablet_copy_client.cc:647] T f4d2b8c005b64bbb8e6621da068d15e5 P 4360b59e3a154688a79ca5f9c0a37bc9: tablet copy: Starting download of 1 WAL segments...
I20240501 14:05:39.563563 9333 tablet_copy_client.cc:308] T 3d6859d53fd747a5a5e94722626fa522 P 19a6d2d8a08142d5bffcde3a37234692: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.130:34377
I20240501 14:05:39.565320 5146 tablet_copy_service.cc:140] P e9c60449cf3c46f8aca4ab6c535302a4: Received BeginTabletCopySession request for tablet 3d6859d53fd747a5a5e94722626fa522 from peer 19a6d2d8a08142d5bffcde3a37234692 ({username='slave'} at 127.29.124.134:41917)
I20240501 14:05:39.565485 5146 tablet_copy_service.cc:161] P e9c60449cf3c46f8aca4ab6c535302a4: Beginning new tablet copy session on tablet 3d6859d53fd747a5a5e94722626fa522 from peer 19a6d2d8a08142d5bffcde3a37234692 at {username='slave'} at 127.29.124.134:41917: session id = 19a6d2d8a08142d5bffcde3a37234692-3d6859d53fd747a5a5e94722626fa522
I20240501 14:05:39.559618 5125 raft_consensus.cc:2932] T b284aa82594a44f5885f32fa05844913 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:39.568421 5146 tablet_copy_source_session.cc:205] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:05:39.569339 9334 tablet_bootstrap.cc:492] T ac4b234f794149e4b6def7a3b358942d P 6dac90f01e744f9c8701a404440a4c77: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:05:39.570147 9334 tablet_bootstrap.cc:492] T ac4b234f794149e4b6def7a3b358942d P 6dac90f01e744f9c8701a404440a4c77: Bootstrap complete.
I20240501 14:05:39.570474 9334 ts_tablet_manager.cc:1367] T ac4b234f794149e4b6def7a3b358942d P 6dac90f01e744f9c8701a404440a4c77: Time spent bootstrapping tablet: real 0.235s user 0.041s sys 0.011s
I20240501 14:05:39.571327 9334 raft_consensus.cc:348] T ac4b234f794149e4b6def7a3b358942d P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } }
I20240501 14:05:39.571601 9334 raft_consensus.cc:729] T ac4b234f794149e4b6def7a3b358942d P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6dac90f01e744f9c8701a404440a4c77, State: Initialized, Role: LEARNER
I20240501 14:05:39.571847 9334 consensus_queue.cc:260] T ac4b234f794149e4b6def7a3b358942d P 6dac90f01e744f9c8701a404440a4c77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } }
I20240501 14:05:39.572479 9334 ts_tablet_manager.cc:1398] T ac4b234f794149e4b6def7a3b358942d P 6dac90f01e744f9c8701a404440a4c77: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 14:05:39.573141 5146 tablet_copy_service.cc:342] P e9c60449cf3c46f8aca4ab6c535302a4: Request end of tablet copy session 6dac90f01e744f9c8701a404440a4c77-ac4b234f794149e4b6def7a3b358942d received from {username='slave'} at 127.29.124.136:51401
I20240501 14:05:39.573364 5146 tablet_copy_service.cc:433] P e9c60449cf3c46f8aca4ab6c535302a4: ending tablet copy session 6dac90f01e744f9c8701a404440a4c77-ac4b234f794149e4b6def7a3b358942d on tablet ac4b234f794149e4b6def7a3b358942d with peer 6dac90f01e744f9c8701a404440a4c77
I20240501 14:05:39.575735 6888 raft_consensus.cc:1203] T c8a961e711114f369ba06840c062b982 P 938a24b0e6a040b7966225af26558ff8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:05:39.576963 6889 raft_consensus.cc:2932] T 9b88c43cb8704583a3e65c5c35c205f6 P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:39.592340 9229 tablet_copy_client.cc:308] T f9cb784f60774ea7b6e70ccfebf867a8 P 19a6d2d8a08142d5bffcde3a37234692: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.130:34377
I20240501 14:05:39.592815 9333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d6859d53fd747a5a5e94722626fa522. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:39.593079 5146 tablet_copy_service.cc:140] P e9c60449cf3c46f8aca4ab6c535302a4: Received BeginTabletCopySession request for tablet f9cb784f60774ea7b6e70ccfebf867a8 from peer 19a6d2d8a08142d5bffcde3a37234692 ({username='slave'} at 127.29.124.134:41917)
I20240501 14:05:39.593372 5146 tablet_copy_service.cc:161] P e9c60449cf3c46f8aca4ab6c535302a4: Beginning new tablet copy session on tablet f9cb784f60774ea7b6e70ccfebf867a8 from peer 19a6d2d8a08142d5bffcde3a37234692 at {username='slave'} at 127.29.124.134:41917: session id = 19a6d2d8a08142d5bffcde3a37234692-f9cb784f60774ea7b6e70ccfebf867a8
I20240501 14:05:39.593979 9382 ts_tablet_manager.cc:902] T 88ed5438fc934adc9fb98c6cb4476598 P 4360b59e3a154688a79ca5f9c0a37bc9: Initiating tablet copy from peer e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377)
I20240501 14:05:39.595975 9281 tablet_copy_client.cc:523] T f4d2b8c005b64bbb8e6621da068d15e5 P 4360b59e3a154688a79ca5f9c0a37bc9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:05:39.596340 5146 tablet_copy_source_session.cc:205] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:05:39.597616 9229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9cb784f60774ea7b6e70ccfebf867a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:39.605216 9333 tablet_copy_client.cc:783] T 3d6859d53fd747a5a5e94722626fa522 P 19a6d2d8a08142d5bffcde3a37234692: tablet copy: Starting download of 0 data blocks...
I20240501 14:05:39.605883 9333 tablet_copy_client.cc:647] T 3d6859d53fd747a5a5e94722626fa522 P 19a6d2d8a08142d5bffcde3a37234692: tablet copy: Starting download of 1 WAL segments...
I20240501 14:05:39.613754 9381 tablet_copy_client.cc:308] T 1a7bbfc15a76450bbc9197037e856e08 P 4360b59e3a154688a79ca5f9c0a37bc9: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.131:33209
I20240501 14:05:39.617214 5276 tablet_copy_service.cc:140] P 1c3fdc5272484ac09ef3e5e36a08f27c: Received BeginTabletCopySession request for tablet 1a7bbfc15a76450bbc9197037e856e08 from peer 4360b59e3a154688a79ca5f9c0a37bc9 ({username='slave'} at 127.29.124.135:51325)
I20240501 14:05:39.617417 5276 tablet_copy_service.cc:161] P 1c3fdc5272484ac09ef3e5e36a08f27c: Beginning new tablet copy session on tablet 1a7bbfc15a76450bbc9197037e856e08 from peer 4360b59e3a154688a79ca5f9c0a37bc9 at {username='slave'} at 127.29.124.135:51325: session id = 4360b59e3a154688a79ca5f9c0a37bc9-1a7bbfc15a76450bbc9197037e856e08
I20240501 14:05:39.620273 5276 tablet_copy_source_session.cc:205] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:05:39.622517 9381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a7bbfc15a76450bbc9197037e856e08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:39.628268 7287 raft_consensus.cc:1203] T ac4b234f794149e4b6def7a3b358942d P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:05:39.633666 9381 tablet_copy_client.cc:783] T 1a7bbfc15a76450bbc9197037e856e08 P 4360b59e3a154688a79ca5f9c0a37bc9: tablet copy: Starting download of 0 data blocks...
I20240501 14:05:39.633970 9381 tablet_copy_client.cc:647] T 1a7bbfc15a76450bbc9197037e856e08 P 4360b59e3a154688a79ca5f9c0a37bc9: tablet copy: Starting download of 1 WAL segments...
I20240501 14:05:39.639189 9381 tablet_copy_client.cc:523] T 1a7bbfc15a76450bbc9197037e856e08 P 4360b59e3a154688a79ca5f9c0a37bc9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:05:39.640858 9281 tablet_bootstrap.cc:492] T f4d2b8c005b64bbb8e6621da068d15e5 P 4360b59e3a154688a79ca5f9c0a37bc9: Bootstrap starting.
I20240501 14:05:39.642704 9381 tablet_bootstrap.cc:492] T 1a7bbfc15a76450bbc9197037e856e08 P 4360b59e3a154688a79ca5f9c0a37bc9: Bootstrap starting.
I20240501 14:05:39.660126 9333 tablet_copy_client.cc:523] T 3d6859d53fd747a5a5e94722626fa522 P 19a6d2d8a08142d5bffcde3a37234692: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:05:39.677840 9227 tablet_copy_client.cc:308] T 63650547ab444f45bb377bcb49f258ad P 938a24b0e6a040b7966225af26558ff8: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.129:44033
I20240501 14:05:39.680234 5015 tablet_copy_service.cc:140] P 04088ee3bea440a7bd5ff772152b9410: Received BeginTabletCopySession request for tablet 63650547ab444f45bb377bcb49f258ad from peer 938a24b0e6a040b7966225af26558ff8 ({username='slave'} at 127.29.124.133:49901)
I20240501 14:05:39.680418 5015 tablet_copy_service.cc:161] P 04088ee3bea440a7bd5ff772152b9410: Beginning new tablet copy session on tablet 63650547ab444f45bb377bcb49f258ad from peer 938a24b0e6a040b7966225af26558ff8 at {username='slave'} at 127.29.124.133:49901: session id = 938a24b0e6a040b7966225af26558ff8-63650547ab444f45bb377bcb49f258ad
I20240501 14:05:39.683375 5015 tablet_copy_source_session.cc:205] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:05:39.684624 9227 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63650547ab444f45bb377bcb49f258ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:39.685333 9229 tablet_copy_client.cc:783] T f9cb784f60774ea7b6e70ccfebf867a8 P 19a6d2d8a08142d5bffcde3a37234692: tablet copy: Starting download of 0 data blocks...
I20240501 14:05:39.685633 9229 tablet_copy_client.cc:647] T f9cb784f60774ea7b6e70ccfebf867a8 P 19a6d2d8a08142d5bffcde3a37234692: tablet copy: Starting download of 1 WAL segments...
I20240501 14:05:39.686774 9333 tablet_bootstrap.cc:492] T 3d6859d53fd747a5a5e94722626fa522 P 19a6d2d8a08142d5bffcde3a37234692: Bootstrap starting.
I20240501 14:05:39.691431 9229 tablet_copy_client.cc:523] T f9cb784f60774ea7b6e70ccfebf867a8 P 19a6d2d8a08142d5bffcde3a37234692: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:05:39.694496 9270 tablet_bootstrap.cc:492] T c8a961e711114f369ba06840c062b982 P 4360b59e3a154688a79ca5f9c0a37bc9: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:05:39.694921 9270 tablet_bootstrap.cc:492] T c8a961e711114f369ba06840c062b982 P 4360b59e3a154688a79ca5f9c0a37bc9: Bootstrap complete.
I20240501 14:05:39.695155 9229 tablet_bootstrap.cc:492] T f9cb784f60774ea7b6e70ccfebf867a8 P 19a6d2d8a08142d5bffcde3a37234692: Bootstrap starting.
I20240501 14:05:39.695264 9270 ts_tablet_manager.cc:1367] T c8a961e711114f369ba06840c062b982 P 4360b59e3a154688a79ca5f9c0a37bc9: Time spent bootstrapping tablet: real 0.481s user 0.051s sys 0.004s
I20240501 14:05:39.700898 9270 raft_consensus.cc:348] T c8a961e711114f369ba06840c062b982 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }
I20240501 14:05:39.701191 9270 raft_consensus.cc:729] T c8a961e711114f369ba06840c062b982 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4360b59e3a154688a79ca5f9c0a37bc9, State: Initialized, Role: LEARNER
I20240501 14:05:39.702491 9270 consensus_queue.cc:260] T c8a961e711114f369ba06840c062b982 P 4360b59e3a154688a79ca5f9c0a37bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }
I20240501 14:05:39.725950 9270 ts_tablet_manager.cc:1398] T c8a961e711114f369ba06840c062b982 P 4360b59e3a154688a79ca5f9c0a37bc9: Time spent starting tablet: real 0.031s user 0.001s sys 0.003s
I20240501 14:05:39.726666 5146 tablet_copy_service.cc:342] P e9c60449cf3c46f8aca4ab6c535302a4: Request end of tablet copy session 4360b59e3a154688a79ca5f9c0a37bc9-c8a961e711114f369ba06840c062b982 received from {username='slave'} at 127.29.124.135:58119
I20240501 14:05:39.727030 5146 tablet_copy_service.cc:433] P e9c60449cf3c46f8aca4ab6c535302a4: ending tablet copy session 4360b59e3a154688a79ca5f9c0a37bc9-c8a961e711114f369ba06840c062b982 on tablet c8a961e711114f369ba06840c062b982 with peer 4360b59e3a154688a79ca5f9c0a37bc9
I20240501 14:05:39.744153 9228 tablet_bootstrap.cc:492] T 99d958617b7e4ef385b94f4c73f4dd84 P 4360b59e3a154688a79ca5f9c0a37bc9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:05:39.745007 9228 tablet_bootstrap.cc:492] T 99d958617b7e4ef385b94f4c73f4dd84 P 4360b59e3a154688a79ca5f9c0a37bc9: Bootstrap complete.
I20240501 14:05:39.748442 9228 ts_tablet_manager.cc:1367] T 99d958617b7e4ef385b94f4c73f4dd84 P 4360b59e3a154688a79ca5f9c0a37bc9: Time spent bootstrapping tablet: real 0.677s user 0.041s sys 0.010s
I20240501 14:05:39.749223 9228 raft_consensus.cc:348] T 99d958617b7e4ef385b94f4c73f4dd84 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } }
I20240501 14:05:39.749605 9228 raft_consensus.cc:729] T 99d958617b7e4ef385b94f4c73f4dd84 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4360b59e3a154688a79ca5f9c0a37bc9, State: Initialized, Role: LEARNER
I20240501 14:05:39.749838 9228 consensus_queue.cc:260] T 99d958617b7e4ef385b94f4c73f4dd84 P 4360b59e3a154688a79ca5f9c0a37bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } }
I20240501 14:05:39.750495 9228 ts_tablet_manager.cc:1398] T 99d958617b7e4ef385b94f4c73f4dd84 P 4360b59e3a154688a79ca5f9c0a37bc9: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:05:39.747205 9382 tablet_copy_client.cc:308] T 88ed5438fc934adc9fb98c6cb4476598 P 4360b59e3a154688a79ca5f9c0a37bc9: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.130:34377
I20240501 14:05:39.751252 5015 tablet_copy_service.cc:342] P 04088ee3bea440a7bd5ff772152b9410: Request end of tablet copy session 4360b59e3a154688a79ca5f9c0a37bc9-99d958617b7e4ef385b94f4c73f4dd84 received from {username='slave'} at 127.29.124.135:54069
I20240501 14:05:39.751405 5015 tablet_copy_service.cc:433] P 04088ee3bea440a7bd5ff772152b9410: ending tablet copy session 4360b59e3a154688a79ca5f9c0a37bc9-99d958617b7e4ef385b94f4c73f4dd84 on tablet 99d958617b7e4ef385b94f4c73f4dd84 with peer 4360b59e3a154688a79ca5f9c0a37bc9
I20240501 14:05:39.751838 5146 tablet_copy_service.cc:140] P e9c60449cf3c46f8aca4ab6c535302a4: Received BeginTabletCopySession request for tablet 88ed5438fc934adc9fb98c6cb4476598 from peer 4360b59e3a154688a79ca5f9c0a37bc9 ({username='slave'} at 127.29.124.135:58119)
I20240501 14:05:39.752002 5146 tablet_copy_service.cc:161] P e9c60449cf3c46f8aca4ab6c535302a4: Beginning new tablet copy session on tablet 88ed5438fc934adc9fb98c6cb4476598 from peer 4360b59e3a154688a79ca5f9c0a37bc9 at {username='slave'} at 127.29.124.135:58119: session id = 4360b59e3a154688a79ca5f9c0a37bc9-88ed5438fc934adc9fb98c6cb4476598
I20240501 14:05:39.753178 7025 raft_consensus.cc:1203] T fee16905387d428aa02cd2070c86c790 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.10->[1.11-1.11] Dedup: 1.11->[]
I20240501 14:05:39.754949 5146 tablet_copy_source_session.cc:205] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:05:39.756778 9382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88ed5438fc934adc9fb98c6cb4476598. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:39.770296 4993 consensus_queue.cc:237] T a726769d24284fedadd8e9e63cd27d05 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: NON_VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { promote: true } }
I20240501 14:05:39.793545 7287 raft_consensus.cc:1203] T 43b46fcf827c4275a47291afb107c290 P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
W20240501 14:05:39.795733 4928 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209): Couldn't send request to peer 1c3fdc5272484ac09ef3e5e36a08f27c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a726769d24284fedadd8e9e63cd27d05. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:39.797310 5385 raft_consensus.cc:1261] T a726769d24284fedadd8e9e63cd27d05 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:39.798089 9146 consensus_queue.cc:1035] T a726769d24284fedadd8e9e63cd27d05 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:39.801609 9382 tablet_copy_client.cc:783] T 88ed5438fc934adc9fb98c6cb4476598 P 4360b59e3a154688a79ca5f9c0a37bc9: tablet copy: Starting download of 0 data blocks...
I20240501 14:05:39.801905 9382 tablet_copy_client.cc:647] T 88ed5438fc934adc9fb98c6cb4476598 P 4360b59e3a154688a79ca5f9c0a37bc9: tablet copy: Starting download of 1 WAL segments...
I20240501 14:05:39.813426 9227 tablet_copy_client.cc:783] T 63650547ab444f45bb377bcb49f258ad P 938a24b0e6a040b7966225af26558ff8: tablet copy: Starting download of 0 data blocks...
I20240501 14:05:39.816056 5125 raft_consensus.cc:1261] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:39.817755 9382 tablet_copy_client.cc:523] T 88ed5438fc934adc9fb98c6cb4476598 P 4360b59e3a154688a79ca5f9c0a37bc9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:05:39.818737 9391 raft_consensus.cc:1050] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf: attempting to promote NON_VOTER 938a24b0e6a040b7966225af26558ff8 to VOTER
I20240501 14:05:39.819563 9391 consensus_queue.cc:237] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:05:39.821399 9382 tablet_bootstrap.cc:492] T 88ed5438fc934adc9fb98c6cb4476598 P 4360b59e3a154688a79ca5f9c0a37bc9: Bootstrap starting.
I20240501 14:05:39.821768 9146 raft_consensus.cc:2932] T a726769d24284fedadd8e9e63cd27d05 P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: NON_VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { promote: true } } }
I20240501 14:05:39.825888 9009 consensus_queue.cc:1035] T a726769d24284fedadd8e9e63cd27d05 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:39.826923 5382 raft_consensus.cc:2932] T a726769d24284fedadd8e9e63cd27d05 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: NON_VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { promote: true } } }
I20240501 14:05:39.833197 4847 catalog_manager.cc:5484] T a726769d24284fedadd8e9e63cd27d05 P 04088ee3bea440a7bd5ff772152b9410 reported cstate change: config changed from index 4 to 5, NON_VOTER 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) added. New cstate: current_term: 1 leader_uuid: "04088ee3bea440a7bd5ff772152b9410" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: NON_VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:05:39.839272 7156 raft_consensus.cc:1203] T c8a961e711114f369ba06840c062b982 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:05:39.840068 5254 raft_consensus.cc:1261] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:39.840683 9390 consensus_queue.cc:1035] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:39.842599 5126 raft_consensus.cc:1261] T 420642af520d45ffbc7f2d35f93e75a7 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:39.843410 9390 consensus_queue.cc:1035] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:39.856568 9381 tablet_bootstrap.cc:492] T 1a7bbfc15a76450bbc9197037e856e08 P 4360b59e3a154688a79ca5f9c0a37bc9: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:05:39.857051 9381 tablet_bootstrap.cc:492] T 1a7bbfc15a76450bbc9197037e856e08 P 4360b59e3a154688a79ca5f9c0a37bc9: Bootstrap complete.
I20240501 14:05:39.857385 9381 ts_tablet_manager.cc:1367] T 1a7bbfc15a76450bbc9197037e856e08 P 4360b59e3a154688a79ca5f9c0a37bc9: Time spent bootstrapping tablet: real 0.215s user 0.051s sys 0.005s
I20240501 14:05:39.858233 9381 raft_consensus.cc:348] T 1a7bbfc15a76450bbc9197037e856e08 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } }
I20240501 14:05:39.871932 7287 raft_consensus.cc:1203] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:05:39.881460 9381 raft_consensus.cc:729] T 1a7bbfc15a76450bbc9197037e856e08 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4360b59e3a154688a79ca5f9c0a37bc9, State: Initialized, Role: LEARNER
I20240501 14:05:39.881837 9381 consensus_queue.cc:260] T 1a7bbfc15a76450bbc9197037e856e08 P 4360b59e3a154688a79ca5f9c0a37bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } }
I20240501 14:05:39.883498 9381 ts_tablet_manager.cc:1398] T 1a7bbfc15a76450bbc9197037e856e08 P 4360b59e3a154688a79ca5f9c0a37bc9: Time spent starting tablet: real 0.026s user 0.002s sys 0.000s
I20240501 14:05:39.884287 5276 tablet_copy_service.cc:342] P 1c3fdc5272484ac09ef3e5e36a08f27c: Request end of tablet copy session 4360b59e3a154688a79ca5f9c0a37bc9-1a7bbfc15a76450bbc9197037e856e08 received from {username='slave'} at 127.29.124.135:51325
I20240501 14:05:39.884436 5276 tablet_copy_service.cc:433] P 1c3fdc5272484ac09ef3e5e36a08f27c: ending tablet copy session 4360b59e3a154688a79ca5f9c0a37bc9-1a7bbfc15a76450bbc9197037e856e08 on tablet 1a7bbfc15a76450bbc9197037e856e08 with peer 4360b59e3a154688a79ca5f9c0a37bc9
W20240501 14:05:39.904534 4930 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:39.908764 5125 raft_consensus.cc:2932] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: NON_VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { promote: true } } }
I20240501 14:05:39.912178 9391 raft_consensus.cc:2932] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:39.915944 9334 ts_tablet_manager.cc:902] T 9b88c43cb8704583a3e65c5c35c205f6 P 6dac90f01e744f9c8701a404440a4c77: Initiating tablet copy from peer be56773105e744919601127a94e98adf (127.29.124.132:35459)
I20240501 14:05:39.919209 5126 raft_consensus.cc:2932] T 420642af520d45ffbc7f2d35f93e75a7 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:39.923240 4847 catalog_manager.cc:5484] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf reported cstate change: config changed from index 4 to 5, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:05:39.924559 5254 raft_consensus.cc:2932] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:39.967080 9334 tablet_copy_client.cc:308] T 9b88c43cb8704583a3e65c5c35c205f6 P 6dac90f01e744f9c8701a404440a4c77: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.132:35459
I20240501 14:05:39.967816 5406 tablet_copy_service.cc:140] P be56773105e744919601127a94e98adf: Received BeginTabletCopySession request for tablet 9b88c43cb8704583a3e65c5c35c205f6 from peer 6dac90f01e744f9c8701a404440a4c77 ({username='slave'} at 127.29.124.136:50719)
I20240501 14:05:39.967993 5406 tablet_copy_service.cc:161] P be56773105e744919601127a94e98adf: Beginning new tablet copy session on tablet 9b88c43cb8704583a3e65c5c35c205f6 from peer 6dac90f01e744f9c8701a404440a4c77 at {username='slave'} at 127.29.124.136:50719: session id = 6dac90f01e744f9c8701a404440a4c77-9b88c43cb8704583a3e65c5c35c205f6
I20240501 14:05:39.838856 6888 raft_consensus.cc:1261] T 420642af520d45ffbc7f2d35f93e75a7 P 938a24b0e6a040b7966225af26558ff8 [term 1 LEARNER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:39.907459 9296 ts_tablet_manager.cc:902] T b284aa82594a44f5885f32fa05844913 P 938a24b0e6a040b7966225af26558ff8: Initiating tablet copy from peer be56773105e744919601127a94e98adf (127.29.124.132:35459)
I20240501 14:05:39.970703 9391 consensus_queue.cc:1035] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:39.971037 5406 tablet_copy_source_session.cc:205] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:05:39.971309 9281 tablet_bootstrap.cc:492] T f4d2b8c005b64bbb8e6621da068d15e5 P 4360b59e3a154688a79ca5f9c0a37bc9: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:05:39.971753 9281 tablet_bootstrap.cc:492] T f4d2b8c005b64bbb8e6621da068d15e5 P 4360b59e3a154688a79ca5f9c0a37bc9: Bootstrap complete.
I20240501 14:05:39.972043 9281 ts_tablet_manager.cc:1367] T f4d2b8c005b64bbb8e6621da068d15e5 P 4360b59e3a154688a79ca5f9c0a37bc9: Time spent bootstrapping tablet: real 0.372s user 0.043s sys 0.005s
I20240501 14:05:39.976215 9334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b88c43cb8704583a3e65c5c35c205f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:39.984333 6888 raft_consensus.cc:2932] T 420642af520d45ffbc7f2d35f93e75a7 P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:39.995983 9227 tablet_copy_client.cc:647] T 63650547ab444f45bb377bcb49f258ad P 938a24b0e6a040b7966225af26558ff8: tablet copy: Starting download of 1 WAL segments...
I20240501 14:05:39.972801 9281 raft_consensus.cc:348] T f4d2b8c005b64bbb8e6621da068d15e5 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } }
I20240501 14:05:39.999204 9281 raft_consensus.cc:729] T f4d2b8c005b64bbb8e6621da068d15e5 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4360b59e3a154688a79ca5f9c0a37bc9, State: Initialized, Role: LEARNER
I20240501 14:05:39.999433 9281 consensus_queue.cc:260] T f4d2b8c005b64bbb8e6621da068d15e5 P 4360b59e3a154688a79ca5f9c0a37bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } }
I20240501 14:05:40.000037 9281 ts_tablet_manager.cc:1398] T f4d2b8c005b64bbb8e6621da068d15e5 P 4360b59e3a154688a79ca5f9c0a37bc9: Time spent starting tablet: real 0.028s user 0.003s sys 0.000s
I20240501 14:05:40.000944 5406 tablet_copy_service.cc:342] P be56773105e744919601127a94e98adf: Request end of tablet copy session 4360b59e3a154688a79ca5f9c0a37bc9-f4d2b8c005b64bbb8e6621da068d15e5 received from {username='slave'} at 127.29.124.135:51309
I20240501 14:05:40.001118 5406 tablet_copy_service.cc:433] P be56773105e744919601127a94e98adf: ending tablet copy session 4360b59e3a154688a79ca5f9c0a37bc9-f4d2b8c005b64bbb8e6621da068d15e5 on tablet f4d2b8c005b64bbb8e6621da068d15e5 with peer 4360b59e3a154688a79ca5f9c0a37bc9
I20240501 14:05:40.007663 9227 tablet_copy_client.cc:523] T 63650547ab444f45bb377bcb49f258ad P 938a24b0e6a040b7966225af26558ff8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:05:40.024101 9227 tablet_bootstrap.cc:492] T 63650547ab444f45bb377bcb49f258ad P 938a24b0e6a040b7966225af26558ff8: Bootstrap starting.
I20240501 14:05:40.033037 9296 tablet_copy_client.cc:308] T b284aa82594a44f5885f32fa05844913 P 938a24b0e6a040b7966225af26558ff8: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.132:35459
I20240501 14:05:40.033759 5406 tablet_copy_service.cc:140] P be56773105e744919601127a94e98adf: Received BeginTabletCopySession request for tablet b284aa82594a44f5885f32fa05844913 from peer 938a24b0e6a040b7966225af26558ff8 ({username='slave'} at 127.29.124.133:50583)
I20240501 14:05:40.033996 5406 tablet_copy_service.cc:161] P be56773105e744919601127a94e98adf: Beginning new tablet copy session on tablet b284aa82594a44f5885f32fa05844913 from peer 938a24b0e6a040b7966225af26558ff8 at {username='slave'} at 127.29.124.133:50583: session id = 938a24b0e6a040b7966225af26558ff8-b284aa82594a44f5885f32fa05844913
I20240501 14:05:40.034798 5385 consensus_queue.cc:237] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } }
I20240501 14:05:40.036723 5406 tablet_copy_source_session.cc:205] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:05:40.037285 5125 raft_consensus.cc:1261] T b284aa82594a44f5885f32fa05844913 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:40.037819 9391 consensus_queue.cc:1035] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:40.038347 9296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b284aa82594a44f5885f32fa05844913. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:40.039985 4993 raft_consensus.cc:1261] T b284aa82594a44f5885f32fa05844913 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:40.040051 9391 raft_consensus.cc:2932] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
W20240501 14:05:40.041968 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b284aa82594a44f5885f32fa05844913. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:40.042562 7156 raft_consensus.cc:1203] T f4d2b8c005b64bbb8e6621da068d15e5 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:05:40.043448 8804 consensus_queue.cc:1035] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:40.045563 9296 tablet_copy_client.cc:783] T b284aa82594a44f5885f32fa05844913 P 938a24b0e6a040b7966225af26558ff8: tablet copy: Starting download of 0 data blocks...
W20240501 14:05:40.046223 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:40.045908 5125 raft_consensus.cc:2932] T b284aa82594a44f5885f32fa05844913 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:40.045820 9296 tablet_copy_client.cc:647] T b284aa82594a44f5885f32fa05844913 P 938a24b0e6a040b7966225af26558ff8: tablet copy: Starting download of 1 WAL segments...
I20240501 14:05:40.048060 9296 tablet_copy_client.cc:523] T b284aa82594a44f5885f32fa05844913 P 938a24b0e6a040b7966225af26558ff8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:05:40.045027 4993 raft_consensus.cc:2932] T b284aa82594a44f5885f32fa05844913 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:40.054559 4848 catalog_manager.cc:5484] T b284aa82594a44f5885f32fa05844913 P 04088ee3bea440a7bd5ff772152b9410 reported cstate change: config changed from index 4 to 5, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) added. New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:40.055305 9296 tablet_bootstrap.cc:492] T b284aa82594a44f5885f32fa05844913 P 938a24b0e6a040b7966225af26558ff8: Bootstrap starting.
I20240501 14:05:40.070973 9382 tablet_bootstrap.cc:492] T 88ed5438fc934adc9fb98c6cb4476598 P 4360b59e3a154688a79ca5f9c0a37bc9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:05:40.071452 9382 tablet_bootstrap.cc:492] T 88ed5438fc934adc9fb98c6cb4476598 P 4360b59e3a154688a79ca5f9c0a37bc9: Bootstrap complete.
I20240501 14:05:40.071724 9382 ts_tablet_manager.cc:1367] T 88ed5438fc934adc9fb98c6cb4476598 P 4360b59e3a154688a79ca5f9c0a37bc9: Time spent bootstrapping tablet: real 0.250s user 0.034s sys 0.018s
I20240501 14:05:40.072526 9382 raft_consensus.cc:348] T 88ed5438fc934adc9fb98c6cb4476598 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } }
I20240501 14:05:40.072796 9382 raft_consensus.cc:729] T 88ed5438fc934adc9fb98c6cb4476598 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4360b59e3a154688a79ca5f9c0a37bc9, State: Initialized, Role: LEARNER
I20240501 14:05:40.073004 9382 consensus_queue.cc:260] T 88ed5438fc934adc9fb98c6cb4476598 P 4360b59e3a154688a79ca5f9c0a37bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } }
I20240501 14:05:40.081409 9051 raft_consensus.cc:1050] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4: attempting to promote NON_VOTER 938a24b0e6a040b7966225af26558ff8 to VOTER
I20240501 14:05:40.082271 9051 consensus_queue.cc:237] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:05:40.092131 9334 tablet_copy_client.cc:783] T 9b88c43cb8704583a3e65c5c35c205f6 P 6dac90f01e744f9c8701a404440a4c77: tablet copy: Starting download of 0 data blocks...
I20240501 14:05:40.092458 9334 tablet_copy_client.cc:647] T 9b88c43cb8704583a3e65c5c35c205f6 P 6dac90f01e744f9c8701a404440a4c77: tablet copy: Starting download of 1 WAL segments...
I20240501 14:05:40.094353 9334 tablet_copy_client.cc:523] T 9b88c43cb8704583a3e65c5c35c205f6 P 6dac90f01e744f9c8701a404440a4c77: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:05:40.095249 7156 raft_consensus.cc:1261] T c8a961e711114f369ba06840c062b982 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 LEARNER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:40.095960 9057 consensus_queue.cc:1035] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:40.098088 9334 tablet_bootstrap.cc:492] T 9b88c43cb8704583a3e65c5c35c205f6 P 6dac90f01e744f9c8701a404440a4c77: Bootstrap starting.
I20240501 14:05:40.098755 6889 raft_consensus.cc:1261] T c8a961e711114f369ba06840c062b982 P 938a24b0e6a040b7966225af26558ff8 [term 1 LEARNER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:40.099649 9237 consensus_queue.cc:1035] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:40.103085 5254 raft_consensus.cc:1261] T c8a961e711114f369ba06840c062b982 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:40.103683 9057 consensus_queue.cc:1035] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:40.114699 4993 raft_consensus.cc:1261] T c8a961e711114f369ba06840c062b982 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:40.115428 9057 consensus_queue.cc:1035] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:40.124042 9057 raft_consensus.cc:2932] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:40.131814 7155 raft_consensus.cc:1203] T 99d958617b7e4ef385b94f4c73f4dd84 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:05:40.130158 6889 raft_consensus.cc:2932] T c8a961e711114f369ba06840c062b982 P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:40.132835 4993 raft_consensus.cc:2932] T c8a961e711114f369ba06840c062b982 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:40.146081 9237 raft_consensus.cc:1050] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4: attempting to promote NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 to VOTER
I20240501 14:05:40.147001 9237 consensus_queue.cc:237] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:05:40.154072 6889 raft_consensus.cc:1261] T c8a961e711114f369ba06840c062b982 P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:05:40.154749 9051 consensus_queue.cc:1035] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:05:40.163439 5255 raft_consensus.cc:1261] T c8a961e711114f369ba06840c062b982 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:05:40.163487 4993 raft_consensus.cc:1261] T c8a961e711114f369ba06840c062b982 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:05:40.164067 9237 consensus_queue.cc:1035] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:05:40.164283 7156 raft_consensus.cc:1261] T c8a961e711114f369ba06840c062b982 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 LEARNER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:05:40.164804 9057 consensus_queue.cc:1035] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:40.165570 9057 consensus_queue.cc:1035] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:40.166657 7154 raft_consensus.cc:2932] T c8a961e711114f369ba06840c062b982 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:40.166844 4993 raft_consensus.cc:2932] T c8a961e711114f369ba06840c062b982 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:40.169569 9057 raft_consensus.cc:2932] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:40.175280 9304 tablet_bootstrap.cc:492] T a726769d24284fedadd8e9e63cd27d05 P 19a6d2d8a08142d5bffcde3a37234692: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:05:40.174871 6889 raft_consensus.cc:2932] T c8a961e711114f369ba06840c062b982 P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:40.175704 9304 tablet_bootstrap.cc:492] T a726769d24284fedadd8e9e63cd27d05 P 19a6d2d8a08142d5bffcde3a37234692: Bootstrap complete.
I20240501 14:05:40.176010 9304 ts_tablet_manager.cc:1367] T a726769d24284fedadd8e9e63cd27d05 P 19a6d2d8a08142d5bffcde3a37234692: Time spent bootstrapping tablet: real 0.809s user 0.044s sys 0.004s
I20240501 14:05:40.176824 9304 raft_consensus.cc:348] T a726769d24284fedadd8e9e63cd27d05 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } }
I20240501 14:05:40.177083 9304 raft_consensus.cc:729] T a726769d24284fedadd8e9e63cd27d05 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 19a6d2d8a08142d5bffcde3a37234692, State: Initialized, Role: LEARNER
I20240501 14:05:40.179080 9237 raft_consensus.cc:1050] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4: attempting to promote NON_VOTER 6dac90f01e744f9c8701a404440a4c77 to VOTER
I20240501 14:05:40.180323 9237 consensus_queue.cc:237] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } }
I20240501 14:05:40.182385 5255 raft_consensus.cc:2932] T c8a961e711114f369ba06840c062b982 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:40.195214 7154 raft_consensus.cc:2932] T c8a961e711114f369ba06840c062b982 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:40.195276 9304 consensus_queue.cc:260] T a726769d24284fedadd8e9e63cd27d05 P 19a6d2d8a08142d5bffcde3a37234692 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } }
I20240501 14:05:40.197897 4854 catalog_manager.cc:5484] T c8a961e711114f369ba06840c062b982 P 938a24b0e6a040b7966225af26558ff8 reported cstate change: config changed from index 5 to 6, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:40.200981 9296 tablet_bootstrap.cc:492] T b284aa82594a44f5885f32fa05844913 P 938a24b0e6a040b7966225af26558ff8: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 1 replicates
I20240501 14:05:40.203177 9296 tablet_bootstrap.cc:492] T b284aa82594a44f5885f32fa05844913 P 938a24b0e6a040b7966225af26558ff8: Bootstrap complete.
I20240501 14:05:40.203436 9296 ts_tablet_manager.cc:1367] T b284aa82594a44f5885f32fa05844913 P 938a24b0e6a040b7966225af26558ff8: Time spent bootstrapping tablet: real 0.148s user 0.036s sys 0.021s
I20240501 14:05:40.204253 9296 raft_consensus.cc:348] T b284aa82594a44f5885f32fa05844913 P 938a24b0e6a040b7966225af26558ff8 [term 1 LEARNER]: Replica starting. Triggering 1 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }
I20240501 14:05:40.204743 9296 raft_consensus.cc:729] T b284aa82594a44f5885f32fa05844913 P 938a24b0e6a040b7966225af26558ff8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 938a24b0e6a040b7966225af26558ff8, State: Initialized, Role: LEARNER
I20240501 14:05:40.205013 9382 ts_tablet_manager.cc:1398] T 88ed5438fc934adc9fb98c6cb4476598 P 4360b59e3a154688a79ca5f9c0a37bc9: Time spent starting tablet: real 0.133s user 0.004s sys 0.000s
I20240501 14:05:40.204957 9296 consensus_queue.cc:260] T b284aa82594a44f5885f32fa05844913 P 938a24b0e6a040b7966225af26558ff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } }
I20240501 14:05:40.205891 5146 tablet_copy_service.cc:342] P e9c60449cf3c46f8aca4ab6c535302a4: Request end of tablet copy session 4360b59e3a154688a79ca5f9c0a37bc9-88ed5438fc934adc9fb98c6cb4476598 received from {username='slave'} at 127.29.124.135:58119
I20240501 14:05:40.206027 5146 tablet_copy_service.cc:433] P e9c60449cf3c46f8aca4ab6c535302a4: ending tablet copy session 4360b59e3a154688a79ca5f9c0a37bc9-88ed5438fc934adc9fb98c6cb4476598 on tablet 88ed5438fc934adc9fb98c6cb4476598 with peer 4360b59e3a154688a79ca5f9c0a37bc9
I20240501 14:05:40.208195 9296 ts_tablet_manager.cc:1398] T b284aa82594a44f5885f32fa05844913 P 938a24b0e6a040b7966225af26558ff8: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20240501 14:05:40.209803 7287 raft_consensus.cc:1261] T ac4b234f794149e4b6def7a3b358942d P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:05:40.210516 9334 tablet_bootstrap.cc:492] T 9b88c43cb8704583a3e65c5c35c205f6 P 6dac90f01e744f9c8701a404440a4c77: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:05:40.210945 9334 tablet_bootstrap.cc:492] T 9b88c43cb8704583a3e65c5c35c205f6 P 6dac90f01e744f9c8701a404440a4c77: Bootstrap complete.
I20240501 14:05:40.210949 9057 consensus_queue.cc:1035] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:05:40.211194 9334 ts_tablet_manager.cc:1367] T 9b88c43cb8704583a3e65c5c35c205f6 P 6dac90f01e744f9c8701a404440a4c77: Time spent bootstrapping tablet: real 0.113s user 0.049s sys 0.008s
I20240501 14:05:40.211937 9334 raft_consensus.cc:348] T 9b88c43cb8704583a3e65c5c35c205f6 P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } }
I20240501 14:05:40.213759 5406 tablet_copy_service.cc:342] P be56773105e744919601127a94e98adf: Request end of tablet copy session 938a24b0e6a040b7966225af26558ff8-b284aa82594a44f5885f32fa05844913 received from {username='slave'} at 127.29.124.133:50583
I20240501 14:05:40.213915 5406 tablet_copy_service.cc:433] P be56773105e744919601127a94e98adf: ending tablet copy session 938a24b0e6a040b7966225af26558ff8-b284aa82594a44f5885f32fa05844913 on tablet b284aa82594a44f5885f32fa05844913 with peer 938a24b0e6a040b7966225af26558ff8
I20240501 14:05:40.216419 7154 raft_consensus.cc:1261] T ac4b234f794149e4b6def7a3b358942d P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:05:40.217584 9057 consensus_queue.cc:1035] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:05:40.230052 5385 raft_consensus.cc:1261] T ac4b234f794149e4b6def7a3b358942d P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:05:40.230829 5382 consensus_queue.cc:237] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } }
I20240501 14:05:40.231245 4850 catalog_manager.cc:5484] T c8a961e711114f369ba06840c062b982 P 4360b59e3a154688a79ca5f9c0a37bc9 reported cstate change: config changed from index 6 to 7, 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:40.233124 9490 consensus_queue.cc:1035] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.003s
I20240501 14:05:40.212172 9334 raft_consensus.cc:729] T 9b88c43cb8704583a3e65c5c35c205f6 P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6dac90f01e744f9c8701a404440a4c77, State: Initialized, Role: LEARNER
I20240501 14:05:40.238830 9334 consensus_queue.cc:260] T 9b88c43cb8704583a3e65c5c35c205f6 P 6dac90f01e744f9c8701a404440a4c77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } }
I20240501 14:05:40.239606 9334 ts_tablet_manager.cc:1398] T 9b88c43cb8704583a3e65c5c35c205f6 P 6dac90f01e744f9c8701a404440a4c77: Time spent starting tablet: real 0.028s user 0.003s sys 0.000s
I20240501 14:05:40.240339 5406 tablet_copy_service.cc:342] P be56773105e744919601127a94e98adf: Request end of tablet copy session 6dac90f01e744f9c8701a404440a4c77-9b88c43cb8704583a3e65c5c35c205f6 received from {username='slave'} at 127.29.124.136:50719
I20240501 14:05:40.240464 5406 tablet_copy_service.cc:433] P be56773105e744919601127a94e98adf: ending tablet copy session 6dac90f01e744f9c8701a404440a4c77-9b88c43cb8704583a3e65c5c35c205f6 on tablet 9b88c43cb8704583a3e65c5c35c205f6 with peer 6dac90f01e744f9c8701a404440a4c77
I20240501 14:05:40.241853 5125 consensus_queue.cc:237] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:05:40.245918 9227 tablet_bootstrap.cc:492] T 63650547ab444f45bb377bcb49f258ad P 938a24b0e6a040b7966225af26558ff8: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:05:40.246140 4993 raft_consensus.cc:1261] T c8a961e711114f369ba06840c062b982 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:05:40.246359 9227 tablet_bootstrap.cc:492] T 63650547ab444f45bb377bcb49f258ad P 938a24b0e6a040b7966225af26558ff8: Bootstrap complete.
I20240501 14:05:40.246681 9227 ts_tablet_manager.cc:1367] T 63650547ab444f45bb377bcb49f258ad P 938a24b0e6a040b7966225af26558ff8: Time spent bootstrapping tablet: real 0.226s user 0.050s sys 0.023s
I20240501 14:05:40.247133 9496 consensus_queue.cc:1035] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:05:40.247473 9227 raft_consensus.cc:348] T 63650547ab444f45bb377bcb49f258ad P 938a24b0e6a040b7966225af26558ff8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }
I20240501 14:05:40.247812 9227 raft_consensus.cc:729] T 63650547ab444f45bb377bcb49f258ad P 938a24b0e6a040b7966225af26558ff8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 938a24b0e6a040b7966225af26558ff8, State: Initialized, Role: LEARNER
I20240501 14:05:40.248044 9227 consensus_queue.cc:260] T 63650547ab444f45bb377bcb49f258ad P 938a24b0e6a040b7966225af26558ff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }
I20240501 14:05:40.248601 9227 ts_tablet_manager.cc:1398] T 63650547ab444f45bb377bcb49f258ad P 938a24b0e6a040b7966225af26558ff8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:05:40.250861 6889 raft_consensus.cc:1261] T c8a961e711114f369ba06840c062b982 P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:05:40.253123 9009 raft_consensus.cc:1050] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410: attempting to promote NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 to VOTER
I20240501 14:05:40.253998 9009 consensus_queue.cc:237] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } }
I20240501 14:05:40.259459 9057 consensus_queue.cc:1035] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:05:40.260324 5015 tablet_copy_service.cc:342] P 04088ee3bea440a7bd5ff772152b9410: Request end of tablet copy session 938a24b0e6a040b7966225af26558ff8-63650547ab444f45bb377bcb49f258ad received from {username='slave'} at 127.29.124.133:49901
I20240501 14:05:40.260453 5015 tablet_copy_service.cc:433] P 04088ee3bea440a7bd5ff772152b9410: ending tablet copy session 938a24b0e6a040b7966225af26558ff8-63650547ab444f45bb377bcb49f258ad on tablet 63650547ab444f45bb377bcb49f258ad with peer 938a24b0e6a040b7966225af26558ff8
I20240501 14:05:40.266280 6888 raft_consensus.cc:2932] T c8a961e711114f369ba06840c062b982 P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:40.269428 5125 raft_consensus.cc:1261] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:05:40.269762 7155 raft_consensus.cc:1261] T fee16905387d428aa02cd2070c86c790 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:05:40.270025 9146 consensus_queue.cc:1035] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:05:40.270668 9057 raft_consensus.cc:2932] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:40.271195 7025 raft_consensus.cc:1261] T fee16905387d428aa02cd2070c86c790 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 LEARNER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:05:40.271761 8375 consensus_queue.cc:1035] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:05:40.272274 8375 consensus_queue.cc:1035] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:05:40.281180 4839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c8a961e711114f369ba06840c062b982 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:05:40.282016 4852 catalog_manager.cc:5484] T c8a961e711114f369ba06840c062b982 P 938a24b0e6a040b7966225af26558ff8 reported cstate change: config changed from index 7 to 8, VOTER 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) evicted. New cstate: current_term: 1 leader_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:40.287395 5236 tablet_service.cc:1514] Processing DeleteTablet for tablet c8a961e711114f369ba06840c062b982 with delete_type TABLET_DATA_TOMBSTONED (TS 1c3fdc5272484ac09ef3e5e36a08f27c not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:39020
I20240501 14:05:40.290812 4993 raft_consensus.cc:2932] T c8a961e711114f369ba06840c062b982 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:40.295367 7154 raft_consensus.cc:1261] T c8a961e711114f369ba06840c062b982 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:05:40.295637 5126 consensus_queue.cc:237] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:05:40.298766 6889 raft_consensus.cc:1261] T c8a961e711114f369ba06840c062b982 P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:05:40.300540 9496 consensus_queue.cc:1035] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:05:40.301358 9496 raft_consensus.cc:1050] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4: attempting to promote NON_VOTER 6dac90f01e744f9c8701a404440a4c77 to VOTER
I20240501 14:05:40.302155 9496 consensus_queue.cc:237] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } }
I20240501 14:05:40.305814 9490 raft_consensus.cc:2932] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:40.307696 4992 raft_consensus.cc:1261] T c8a961e711114f369ba06840c062b982 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:05:40.310004 6889 raft_consensus.cc:2932] T c8a961e711114f369ba06840c062b982 P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:40.315186 4839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c8a961e711114f369ba06840c062b982 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:05:40.309248 9490 consensus_queue.cc:1035] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.011s
I20240501 14:05:40.322389 4854 catalog_manager.cc:5484] T c8a961e711114f369ba06840c062b982 P 938a24b0e6a040b7966225af26558ff8 reported cstate change: config changed from index 8 to 9, VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) evicted. New cstate: current_term: 1 leader_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:40.325389 7285 raft_consensus.cc:1261] T 43b46fcf827c4275a47291afb107c290 P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:05:40.326454 9496 consensus_queue.cc:1035] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:05:40.327347 5386 raft_consensus.cc:1261] T 43b46fcf827c4275a47291afb107c290 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:05:40.329005 9237 consensus_queue.cc:1035] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:05:40.344556 7136 tablet_service.cc:1514] Processing DeleteTablet for tablet c8a961e711114f369ba06840c062b982 with delete_type TABLET_DATA_TOMBSTONED (TS 4360b59e3a154688a79ca5f9c0a37bc9 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:51170
I20240501 14:05:40.344720 7025 raft_consensus.cc:1261] T 43b46fcf827c4275a47291afb107c290 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:05:40.345746 9051 consensus_queue.cc:1035] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:05:40.351660 7154 raft_consensus.cc:1203] T 1a7bbfc15a76450bbc9197037e856e08 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:05:40.358453 9508 tablet_replica.cc:330] T c8a961e711114f369ba06840c062b982 P 1c3fdc5272484ac09ef3e5e36a08f27c: stopping tablet replica
I20240501 14:05:40.358744 9508 raft_consensus.cc:2229] T c8a961e711114f369ba06840c062b982 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:05:40.359033 9508 pending_rounds.cc:62] T c8a961e711114f369ba06840c062b982 P 1c3fdc5272484ac09ef3e5e36a08f27c: Trying to abort 1 pending ops.
I20240501 14:05:40.358937 5386 raft_consensus.cc:2932] T ac4b234f794149e4b6def7a3b358942d P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:40.359153 9508 pending_rounds.cc:68] T c8a961e711114f369ba06840c062b982 P 1c3fdc5272484ac09ef3e5e36a08f27c: Aborting op as it isn't in flight: id { term: 1 index: 7 } timestamp: 7022888305239719936 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "c8a961e711114f369ba06840c062b982" old_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } } new_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } } }
I20240501 14:05:40.359593 9508 raft_consensus.cc:2894] T c8a961e711114f369ba06840c062b982 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Aborting config change with OpId 1.7: Aborted: Op aborted
I20240501 14:05:40.359692 9508 raft_consensus.cc:2866] T c8a961e711114f369ba06840c062b982 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240501 14:05:40.359779 9508 raft_consensus.cc:2258] T c8a961e711114f369ba06840c062b982 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:05:40.363374 9057 raft_consensus.cc:2932] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:40.365451 4993 raft_consensus.cc:2932] T c8a961e711114f369ba06840c062b982 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:40.371017 4852 catalog_manager.cc:5484] T ac4b234f794149e4b6def7a3b358942d P be56773105e744919601127a94e98adf reported cstate change: config changed from index 7 to 8, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:40.373895 9237 raft_consensus.cc:2932] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:40.381419 6889 raft_consensus.cc:1203] T 63650547ab444f45bb377bcb49f258ad P 938a24b0e6a040b7966225af26558ff8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:05:40.378615 7025 raft_consensus.cc:2932] T 43b46fcf827c4275a47291afb107c290 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:40.387555 9508 ts_tablet_manager.cc:1875] T c8a961e711114f369ba06840c062b982 P 1c3fdc5272484ac09ef3e5e36a08f27c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:05:40.387663 5386 raft_consensus.cc:2932] T 43b46fcf827c4275a47291afb107c290 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:40.390422 7024 raft_consensus.cc:1261] T f4d2b8c005b64bbb8e6621da068d15e5 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:05:40.391196 7156 raft_consensus.cc:1261] T f4d2b8c005b64bbb8e6621da068d15e5 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 LEARNER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:05:40.391498 9391 consensus_queue.cc:1035] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:05:40.391777 9390 consensus_queue.cc:1035] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:05:40.395205 5254 raft_consensus.cc:1261] T f4d2b8c005b64bbb8e6621da068d15e5 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:05:40.395224 8375 raft_consensus.cc:2932] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:40.397545 8804 consensus_queue.cc:1035] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:05:40.399133 8897 raft_consensus.cc:1050] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c: attempting to promote NON_VOTER 6dac90f01e744f9c8701a404440a4c77 to VOTER
I20240501 14:05:40.399947 8897 consensus_queue.cc:237] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } }
I20240501 14:05:40.400730 7023 raft_consensus.cc:2932] T fee16905387d428aa02cd2070c86c790 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:40.406877 9391 raft_consensus.cc:2932] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:40.408136 5254 raft_consensus.cc:2932] T f4d2b8c005b64bbb8e6621da068d15e5 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
W20240501 14:05:40.409973 5322 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f4d2b8c005b64bbb8e6621da068d15e5. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:40.411911 7024 raft_consensus.cc:2932] T f4d2b8c005b64bbb8e6621da068d15e5 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:40.416543 9304 ts_tablet_manager.cc:1398] T a726769d24284fedadd8e9e63cd27d05 P 19a6d2d8a08142d5bffcde3a37234692: Time spent starting tablet: real 0.240s user 0.004s sys 0.000s
I20240501 14:05:40.417156 5015 tablet_copy_service.cc:342] P 04088ee3bea440a7bd5ff772152b9410: Request end of tablet copy session 19a6d2d8a08142d5bffcde3a37234692-a726769d24284fedadd8e9e63cd27d05 received from {username='slave'} at 127.29.124.134:36629
I20240501 14:05:40.417323 5015 tablet_copy_service.cc:433] P 04088ee3bea440a7bd5ff772152b9410: ending tablet copy session 19a6d2d8a08142d5bffcde3a37234692-a726769d24284fedadd8e9e63cd27d05 on tablet a726769d24284fedadd8e9e63cd27d05 with peer 19a6d2d8a08142d5bffcde3a37234692
I20240501 14:05:40.418579 4849 catalog_manager.cc:5484] T 43b46fcf827c4275a47291afb107c290 P be56773105e744919601127a94e98adf reported cstate change: config changed from index 7 to 8, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:40.427331 4849 catalog_manager.cc:5484] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf reported cstate change: config changed from index 7 to 8, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:05:40.440097 4847 catalog_manager.cc:5484] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 reported cstate change: config changed from index 11 to 12, 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "04088ee3bea440a7bd5ff772152b9410" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:05:40.441409 4973 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:40.442406 7004 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:40.453060 9522 ts_tablet_manager.cc:902] T a726769d24284fedadd8e9e63cd27d05 P 1c3fdc5272484ac09ef3e5e36a08f27c: Initiating tablet copy from peer 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129:44033)
I20240501 14:05:40.458235 7152 raft_consensus.cc:2932] T fee16905387d428aa02cd2070c86c790 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:40.464022 7154 raft_consensus.cc:2932] T ac4b234f794149e4b6def7a3b358942d P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
W20240501 14:05:40.466467 9507 log.cc:924] Time spent T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4: Append to log took a long time: real 0.111s user 0.000s sys 0.000s
I20240501 14:05:40.472096 5125 raft_consensus.cc:2932] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:40.482254 7155 raft_consensus.cc:1203] T 88ed5438fc934adc9fb98c6cb4476598 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240501 14:05:40.392288 7285 raft_consensus.cc:2932] T 43b46fcf827c4275a47291afb107c290 P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:40.483390 7287 raft_consensus.cc:1203] T 9b88c43cb8704583a3e65c5c35c205f6 P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240501 14:05:40.485473 9382 ts_tablet_manager.cc:902] T b284aa82594a44f5885f32fa05844913 P 4360b59e3a154688a79ca5f9c0a37bc9: Initiating tablet copy from peer be56773105e744919601127a94e98adf (127.29.124.132:35459)
I20240501 14:05:40.496547 9508 ts_tablet_manager.cc:1888] T c8a961e711114f369ba06840c062b982 P 1c3fdc5272484ac09ef3e5e36a08f27c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240501 14:05:40.496941 9508 log.cc:1196] T c8a961e711114f369ba06840c062b982 P 1c3fdc5272484ac09ef3e5e36a08f27c: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/wal/wals/c8a961e711114f369ba06840c062b982
I20240501 14:05:40.496940 9515 tablet_replica.cc:330] T c8a961e711114f369ba06840c062b982 P 4360b59e3a154688a79ca5f9c0a37bc9: stopping tablet replica
I20240501 14:05:40.497148 9515 raft_consensus.cc:2229] T c8a961e711114f369ba06840c062b982 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:05:40.498653 9515 raft_consensus.cc:2258] T c8a961e711114f369ba06840c062b982 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:05:40.506035 9515 ts_tablet_manager.cc:1875] T c8a961e711114f369ba06840c062b982 P 4360b59e3a154688a79ca5f9c0a37bc9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:05:40.507467 7287 raft_consensus.cc:1261] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:05:40.508193 8277 consensus_queue.cc:1035] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:05:40.511821 9391 raft_consensus.cc:1050] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf: attempting to promote NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 to VOTER
I20240501 14:05:40.512763 9391 consensus_queue.cc:237] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } }
I20240501 14:05:40.515331 5255 raft_consensus.cc:1261] T f4d2b8c005b64bbb8e6621da068d15e5 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:05:40.515966 9390 consensus_queue.cc:1035] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240501 14:05:40.516311 5322 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f4d2b8c005b64bbb8e6621da068d15e5. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:40.516991 7025 raft_consensus.cc:1261] T f4d2b8c005b64bbb8e6621da068d15e5 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:05:40.517525 7155 raft_consensus.cc:1261] T f4d2b8c005b64bbb8e6621da068d15e5 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 LEARNER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:05:40.518931 7286 raft_consensus.cc:2932] T ac4b234f794149e4b6def7a3b358942d P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:40.519829 8891 consensus_queue.cc:1035] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:05:40.520280 8891 consensus_queue.cc:1035] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:05:40.523447 9390 raft_consensus.cc:2932] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:40.524119 7155 raft_consensus.cc:2932] T f4d2b8c005b64bbb8e6621da068d15e5 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 LEARNER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:40.527045 5125 consensus_queue.cc:237] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } }
I20240501 14:05:40.528409 7025 raft_consensus.cc:1261] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:05:40.529103 7286 raft_consensus.cc:1261] T ac4b234f794149e4b6def7a3b358942d P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:05:40.529295 7153 raft_consensus.cc:1261] T ac4b234f794149e4b6def7a3b358942d P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:05:40.529842 9490 consensus_queue.cc:1035] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:05:40.530026 8277 consensus_queue.cc:1035] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:05:40.530470 9237 consensus_queue.cc:1035] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:05:40.534014 7153 raft_consensus.cc:2932] T ac4b234f794149e4b6def7a3b358942d P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER be56773105e744919601127a94e98adf (127.29.124.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:40.534103 7024 raft_consensus.cc:2932] T f4d2b8c005b64bbb8e6621da068d15e5 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:40.533601 9521 raft_consensus.cc:2932] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER be56773105e744919601127a94e98adf (127.29.124.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:40.538055 7286 raft_consensus.cc:2932] T ac4b234f794149e4b6def7a3b358942d P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER be56773105e744919601127a94e98adf (127.29.124.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:40.547989 6889 raft_consensus.cc:1203] T b284aa82594a44f5885f32fa05844913 P 938a24b0e6a040b7966225af26558ff8 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240501 14:05:40.548287 6889 raft_consensus.cc:2932] T b284aa82594a44f5885f32fa05844913 P 938a24b0e6a040b7966225af26558ff8 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:40.558290 4849 catalog_manager.cc:5484] T ac4b234f794149e4b6def7a3b358942d P 6dac90f01e744f9c8701a404440a4c77 reported cstate change: config changed from index 8 to 9, VOTER be56773105e744919601127a94e98adf (127.29.124.132) evicted. New cstate: current_term: 1 leader_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:40.567498 9009 raft_consensus.cc:1050] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410: attempting to promote NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 to VOTER
I20240501 14:05:40.568312 9009 consensus_queue.cc:237] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } }
I20240501 14:05:40.583757 7286 raft_consensus.cc:2932] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:40.586639 4854 catalog_manager.cc:5484] T f4d2b8c005b64bbb8e6621da068d15e5 P 19a6d2d8a08142d5bffcde3a37234692 reported cstate change: config changed from index 8 to 9, 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:40.586840 8897 raft_consensus.cc:2932] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:40.587774 4993 consensus_queue.cc:237] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }
I20240501 14:05:40.591262 5365 tablet_service.cc:1514] Processing DeleteTablet for tablet ac4b234f794149e4b6def7a3b358942d with delete_type TABLET_DATA_TOMBSTONED (TS be56773105e744919601127a94e98adf not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:43020
I20240501 14:05:40.595194 7025 raft_consensus.cc:2932] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:40.610664 7153 raft_consensus.cc:1261] T 99d958617b7e4ef385b94f4c73f4dd84 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 LEARNER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:40.611800 9146 consensus_queue.cc:1035] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:40.614526 7156 raft_consensus.cc:2932] T f4d2b8c005b64bbb8e6621da068d15e5 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:40.620059 5125 raft_consensus.cc:1261] T 23ca2bf1e59f44fcac8a48aff8d056b1 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:05:40.620687 5255 raft_consensus.cc:2932] T f4d2b8c005b64bbb8e6621da068d15e5 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:40.621642 4838 catalog_manager.cc:4830] TS 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209): tablet c8a961e711114f369ba06840c062b982 (table rebalance_test_table_1 [id=403fdb578f8e4066a56f685925e83936]) successfully deleted
I20240501 14:05:40.623771 4851 catalog_manager.cc:5484] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c reported cstate change: config changed from index 7 to 8, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:05:40.625794 5255 raft_consensus.cc:1261] T 99d958617b7e4ef385b94f4c73f4dd84 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:40.626279 5386 raft_consensus.cc:1261] T 99d958617b7e4ef385b94f4c73f4dd84 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:40.626343 9146 consensus_queue.cc:1035] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:40.627327 9146 consensus_queue.cc:1035] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:40.631753 9009 raft_consensus.cc:2932] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:40.632395 5386 raft_consensus.cc:2932] T 99d958617b7e4ef385b94f4c73f4dd84 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:40.630558 5385 consensus_queue.cc:237] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } }
I20240501 14:05:40.639216 4847 catalog_manager.cc:5484] T 99d958617b7e4ef385b94f4c73f4dd84 P be56773105e744919601127a94e98adf reported cstate change: config changed from index 4 to 5, 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "04088ee3bea440a7bd5ff772152b9410" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:40.640120 8897 consensus_queue.cc:1035] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:05:40.643790 7025 raft_consensus.cc:1261] T 8cc5eec4600640f7a293877c825f8d96 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:05:40.644145 5382 consensus_queue.cc:237] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:05:40.644585 8375 consensus_queue.cc:1035] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240501 14:05:40.645852 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8cc5eec4600640f7a293877c825f8d96. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:40.647323 7024 raft_consensus.cc:1261] T f4d2b8c005b64bbb8e6621da068d15e5 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:05:40.648259 5254 raft_consensus.cc:1261] T f4d2b8c005b64bbb8e6621da068d15e5 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:05:40.648361 5255 raft_consensus.cc:1261] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:40.648809 5126 raft_consensus.cc:1261] T 8cc5eec4600640f7a293877c825f8d96 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:05:40.649031 9390 consensus_queue.cc:1035] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:05:40.649503 8891 consensus_queue.cc:1035] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:40.649506 9390 consensus_queue.cc:1035] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.001s
I20240501 14:05:40.651989 9146 consensus_queue.cc:1035] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:05:40.653682 8375 raft_consensus.cc:2932] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:40.653843 9390 raft_consensus.cc:2932] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:40.654316 7025 raft_consensus.cc:2932] T 8cc5eec4600640f7a293877c825f8d96 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:40.658149 5254 raft_consensus.cc:2932] T f4d2b8c005b64bbb8e6621da068d15e5 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:40.663208 9538 raft_consensus.cc:2932] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:40.664558 4839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f4d2b8c005b64bbb8e6621da068d15e5 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:05:40.665915 7025 raft_consensus.cc:2932] T f4d2b8c005b64bbb8e6621da068d15e5 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:40.667518 5255 raft_consensus.cc:2932] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:40.668383 9550 tablet_replica.cc:330] T ac4b234f794149e4b6def7a3b358942d P be56773105e744919601127a94e98adf: stopping tablet replica
I20240501 14:05:40.668221 4847 catalog_manager.cc:5484] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf reported cstate change: config changed from index 9 to 10, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) evicted. New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:05:40.672259 6889 raft_consensus.cc:1261] T 420642af520d45ffbc7f2d35f93e75a7 P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:40.672587 9550 raft_consensus.cc:2229] T ac4b234f794149e4b6def7a3b358942d P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:05:40.672849 9390 consensus_queue.cc:1035] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:40.673463 9550 raft_consensus.cc:2258] T ac4b234f794149e4b6def7a3b358942d P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:05:40.675949 7267 tablet_service.cc:1514] Processing DeleteTablet for tablet f4d2b8c005b64bbb8e6621da068d15e5 with delete_type TABLET_DATA_TOMBSTONED (TS 6dac90f01e744f9c8701a404440a4c77 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:52244
I20240501 14:05:40.691507 9550 ts_tablet_manager.cc:1875] T ac4b234f794149e4b6def7a3b358942d P be56773105e744919601127a94e98adf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:05:40.693512 6889 raft_consensus.cc:2932] T 420642af520d45ffbc7f2d35f93e75a7 P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:40.701375 7155 raft_consensus.cc:1261] T f4d2b8c005b64bbb8e6621da068d15e5 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240501 14:05:40.702629 4841 catalog_manager.cc:4797] TS 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): delete failed for tablet f4d2b8c005b64bbb8e6621da068d15e5 because the tablet was not found. No further retry: Not found: Tablet not found: f4d2b8c005b64bbb8e6621da068d15e5
I20240501 14:05:40.703737 9391 consensus_queue.cc:1035] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:05:40.706012 7155 raft_consensus.cc:2932] T f4d2b8c005b64bbb8e6621da068d15e5 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:40.711524 5125 raft_consensus.cc:2932] T 23ca2bf1e59f44fcac8a48aff8d056b1 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:40.732254 5236 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:40.737920 9515 ts_tablet_manager.cc:1888] T c8a961e711114f369ba06840c062b982 P 4360b59e3a154688a79ca5f9c0a37bc9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.7
I20240501 14:05:40.738154 9515 log.cc:1196] T c8a961e711114f369ba06840c062b982 P 4360b59e3a154688a79ca5f9c0a37bc9: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/wal/wals/c8a961e711114f369ba06840c062b982
I20240501 14:05:40.739101 4840 catalog_manager.cc:4830] TS 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): tablet c8a961e711114f369ba06840c062b982 (table rebalance_test_table_1 [id=403fdb578f8e4066a56f685925e83936]) successfully deleted
I20240501 14:05:40.742280 8897 raft_consensus.cc:1050] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c: attempting to promote NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 to VOTER
I20240501 14:05:40.743180 8897 consensus_queue.cc:237] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } }
I20240501 14:05:40.743772 5254 raft_consensus.cc:2932] T 99d958617b7e4ef385b94f4c73f4dd84 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:40.752157 4847 catalog_manager.cc:5484] T 8cc5eec4600640f7a293877c825f8d96 P 19a6d2d8a08142d5bffcde3a37234692 reported cstate change: config changed from index 8 to 9, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New cstate: current_term: 1 leader_uuid: "04088ee3bea440a7bd5ff772152b9410" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:40.754266 5255 consensus_queue.cc:237] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } }
I20240501 14:05:40.757985 4839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ac4b234f794149e4b6def7a3b358942d with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:05:40.762574 7285 raft_consensus.cc:1261] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:05:40.763226 7025 raft_consensus.cc:1261] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:05:40.763202 9194 consensus_queue.cc:1035] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:05:40.763803 9194 consensus_queue.cc:1035] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:05:40.766831 9194 raft_consensus.cc:2932] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:40.768838 7025 raft_consensus.cc:2932] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:40.774729 5126 raft_consensus.cc:2932] T 8cc5eec4600640f7a293877c825f8d96 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:40.779665 4847 catalog_manager.cc:5484] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 19a6d2d8a08142d5bffcde3a37234692 reported cstate change: config changed from index 8 to 9, VOTER e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130) evicted. New cstate: current_term: 1 leader_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:40.783385 7285 raft_consensus.cc:2932] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:40.787796 5106 tablet_service.cc:1514] Processing DeleteTablet for tablet 23ca2bf1e59f44fcac8a48aff8d056b1 with delete_type TABLET_DATA_TOMBSTONED (TS e9c60449cf3c46f8aca4ab6c535302a4 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:41850
I20240501 14:05:40.788172 4993 consensus_queue.cc:237] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } }
I20240501 14:05:40.789366 9550 ts_tablet_manager.cc:1888] T ac4b234f794149e4b6def7a3b358942d P be56773105e744919601127a94e98adf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 14:05:40.794622 7153 raft_consensus.cc:2932] T 99d958617b7e4ef385b94f4c73f4dd84 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:40.796990 4838 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 23ca2bf1e59f44fcac8a48aff8d056b1 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:05:40.805647 5385 raft_consensus.cc:1261] T 99d958617b7e4ef385b94f4c73f4dd84 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:40.806571 9009 consensus_queue.cc:1035] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:40.814826 9550 log.cc:1196] T ac4b234f794149e4b6def7a3b358942d P be56773105e744919601127a94e98adf: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/wal/wals/ac4b234f794149e4b6def7a3b358942d
I20240501 14:05:40.816648 9146 raft_consensus.cc:2932] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:40.817701 4839 catalog_manager.cc:4830] TS be56773105e744919601127a94e98adf (127.29.124.132:35459): tablet ac4b234f794149e4b6def7a3b358942d (table rebalance_test_table_6 [id=3c1513e944ee4b518373c4efa5e200cc]) successfully deleted
I20240501 14:05:40.820070 4839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 99d958617b7e4ef385b94f4c73f4dd84 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:05:40.826699 5385 raft_consensus.cc:2932] T 99d958617b7e4ef385b94f4c73f4dd84 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:40.833062 4853 catalog_manager.cc:5484] T 99d958617b7e4ef385b94f4c73f4dd84 P be56773105e744919601127a94e98adf reported cstate change: config changed from index 5 to 6, VOTER 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) evicted. New cstate: current_term: 1 leader_uuid: "04088ee3bea440a7bd5ff772152b9410" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:40.834465 9522 tablet_copy_client.cc:308] T a726769d24284fedadd8e9e63cd27d05 P 1c3fdc5272484ac09ef3e5e36a08f27c: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.129:44033
I20240501 14:05:40.834790 9382 tablet_copy_client.cc:308] T b284aa82594a44f5885f32fa05844913 P 4360b59e3a154688a79ca5f9c0a37bc9: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.132:35459
I20240501 14:05:40.835242 9577 tablet_replica.cc:330] T 23ca2bf1e59f44fcac8a48aff8d056b1 P e9c60449cf3c46f8aca4ab6c535302a4: stopping tablet replica
I20240501 14:05:40.835460 9577 raft_consensus.cc:2229] T 23ca2bf1e59f44fcac8a48aff8d056b1 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:05:40.835762 9577 raft_consensus.cc:2258] T 23ca2bf1e59f44fcac8a48aff8d056b1 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:05:40.835752 5406 tablet_copy_service.cc:140] P be56773105e744919601127a94e98adf: Received BeginTabletCopySession request for tablet b284aa82594a44f5885f32fa05844913 from peer 4360b59e3a154688a79ca5f9c0a37bc9 ({username='slave'} at 127.29.124.135:51309)
I20240501 14:05:40.835901 5406 tablet_copy_service.cc:161] P be56773105e744919601127a94e98adf: Beginning new tablet copy session on tablet b284aa82594a44f5885f32fa05844913 from peer 4360b59e3a154688a79ca5f9c0a37bc9 at {username='slave'} at 127.29.124.135:51309: session id = 4360b59e3a154688a79ca5f9c0a37bc9-b284aa82594a44f5885f32fa05844913
W20240501 14:05:40.836396 5191 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.124.134:40665, user_credentials={real_user=slave}} blocked reactor thread for 60720.3us
I20240501 14:05:40.836827 5015 tablet_copy_service.cc:140] P 04088ee3bea440a7bd5ff772152b9410: Received BeginTabletCopySession request for tablet a726769d24284fedadd8e9e63cd27d05 from peer 1c3fdc5272484ac09ef3e5e36a08f27c ({username='slave'} at 127.29.124.131:58573)
I20240501 14:05:40.837002 5015 tablet_copy_service.cc:161] P 04088ee3bea440a7bd5ff772152b9410: Beginning new tablet copy session on tablet a726769d24284fedadd8e9e63cd27d05 from peer 1c3fdc5272484ac09ef3e5e36a08f27c at {username='slave'} at 127.29.124.131:58573: session id = 1c3fdc5272484ac09ef3e5e36a08f27c-a726769d24284fedadd8e9e63cd27d05
I20240501 14:05:40.838374 9577 ts_tablet_manager.cc:1875] T 23ca2bf1e59f44fcac8a48aff8d056b1 P e9c60449cf3c46f8aca4ab6c535302a4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:05:40.838594 5406 tablet_copy_source_session.cc:205] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:05:40.839635 5015 tablet_copy_source_session.cc:205] T a726769d24284fedadd8e9e63cd27d05 P 04088ee3bea440a7bd5ff772152b9410: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:05:40.841779 5236 tablet_service.cc:1514] Processing DeleteTablet for tablet 99d958617b7e4ef385b94f4c73f4dd84 with delete_type TABLET_DATA_TOMBSTONED (TS 1c3fdc5272484ac09ef3e5e36a08f27c not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:39020
I20240501 14:05:40.842121 9508 tablet_replica.cc:330] T 99d958617b7e4ef385b94f4c73f4dd84 P 1c3fdc5272484ac09ef3e5e36a08f27c: stopping tablet replica
I20240501 14:05:40.842310 9508 raft_consensus.cc:2229] T 99d958617b7e4ef385b94f4c73f4dd84 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:05:40.842715 9508 raft_consensus.cc:2258] T 99d958617b7e4ef385b94f4c73f4dd84 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:05:40.843675 7025 raft_consensus.cc:1261] T 1a7bbfc15a76450bbc9197037e856e08 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:05:40.843767 9508 ts_tablet_manager.cc:1875] T 99d958617b7e4ef385b94f4c73f4dd84 P 1c3fdc5272484ac09ef3e5e36a08f27c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:05:40.844563 7155 raft_consensus.cc:1261] T 1a7bbfc15a76450bbc9197037e856e08 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 LEARNER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:05:40.845245 8277 consensus_queue.cc:1035] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:05:40.845930 8277 consensus_queue.cc:1035] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:05:40.846536 5385 consensus_queue.cc:237] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } }
I20240501 14:05:40.847265 9382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b284aa82594a44f5885f32fa05844913. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:40.851039 9508 ts_tablet_manager.cc:1888] T 99d958617b7e4ef385b94f4c73f4dd84 P 1c3fdc5272484ac09ef3e5e36a08f27c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:05:40.851449 5382 raft_consensus.cc:1261] T 1a7bbfc15a76450bbc9197037e856e08 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:05:40.852386 9508 log.cc:1196] T 99d958617b7e4ef385b94f4c73f4dd84 P 1c3fdc5272484ac09ef3e5e36a08f27c: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/wal/wals/99d958617b7e4ef385b94f4c73f4dd84
I20240501 14:05:40.852821 4839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 420642af520d45ffbc7f2d35f93e75a7 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:05:40.853384 9572 consensus_queue.cc:1035] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:05:40.854343 9522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a726769d24284fedadd8e9e63cd27d05. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:40.857151 4838 catalog_manager.cc:4830] TS 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209): tablet 99d958617b7e4ef385b94f4c73f4dd84 (table rebalance_test_table_8 [id=f26ba2f4e5654a5fa72021df5929200d]) successfully deleted
I20240501 14:05:40.858299 9229 tablet_bootstrap.cc:492] T f9cb784f60774ea7b6e70ccfebf867a8 P 19a6d2d8a08142d5bffcde3a37234692: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:05:40.858774 9229 tablet_bootstrap.cc:492] T f9cb784f60774ea7b6e70ccfebf867a8 P 19a6d2d8a08142d5bffcde3a37234692: Bootstrap complete.
I20240501 14:05:40.859050 9229 ts_tablet_manager.cc:1367] T f9cb784f60774ea7b6e70ccfebf867a8 P 19a6d2d8a08142d5bffcde3a37234692: Time spent bootstrapping tablet: real 1.164s user 0.050s sys 0.003s
I20240501 14:05:40.862061 9229 raft_consensus.cc:348] T f9cb784f60774ea7b6e70ccfebf867a8 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } }
I20240501 14:05:40.862438 9229 raft_consensus.cc:729] T f9cb784f60774ea7b6e70ccfebf867a8 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 19a6d2d8a08142d5bffcde3a37234692, State: Initialized, Role: LEARNER
I20240501 14:05:40.862713 9229 consensus_queue.cc:260] T f9cb784f60774ea7b6e70ccfebf867a8 P 19a6d2d8a08142d5bffcde3a37234692 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } }
W20240501 14:05:40.863773 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2c39b53867e047faaf24cc6f87a8100c. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:40.864239 7285 raft_consensus.cc:1261] T 2c39b53867e047faaf24cc6f87a8100c P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:05:40.864768 9390 consensus_queue.cc:1035] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:05:40.867549 9229 ts_tablet_manager.cc:1398] T f9cb784f60774ea7b6e70ccfebf867a8 P 19a6d2d8a08142d5bffcde3a37234692: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240501 14:05:40.868990 5146 tablet_copy_service.cc:342] P e9c60449cf3c46f8aca4ab6c535302a4: Request end of tablet copy session 19a6d2d8a08142d5bffcde3a37234692-f9cb784f60774ea7b6e70ccfebf867a8 received from {username='slave'} at 127.29.124.134:41917
I20240501 14:05:40.869148 5146 tablet_copy_service.cc:433] P e9c60449cf3c46f8aca4ab6c535302a4: ending tablet copy session 19a6d2d8a08142d5bffcde3a37234692-f9cb784f60774ea7b6e70ccfebf867a8 on tablet f9cb784f60774ea7b6e70ccfebf867a8 with peer 19a6d2d8a08142d5bffcde3a37234692
I20240501 14:05:40.875567 9577 ts_tablet_manager.cc:1888] T 23ca2bf1e59f44fcac8a48aff8d056b1 P e9c60449cf3c46f8aca4ab6c535302a4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 14:05:40.875705 9577 log.cc:1196] T 23ca2bf1e59f44fcac8a48aff8d056b1 P e9c60449cf3c46f8aca4ab6c535302a4: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/wal/wals/23ca2bf1e59f44fcac8a48aff8d056b1
I20240501 14:05:40.875757 7024 raft_consensus.cc:2932] T a726769d24284fedadd8e9e63cd27d05 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: NON_VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { promote: true } } }
I20240501 14:05:40.877360 8277 raft_consensus.cc:2932] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:40.880635 4839 catalog_manager.cc:4830] TS e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377): tablet 23ca2bf1e59f44fcac8a48aff8d056b1 (table rebalance_test_table_7 [id=763c884aecd640b49c65c692fbde9422]) successfully deleted
I20240501 14:05:40.882647 7024 raft_consensus.cc:2932] T 1a7bbfc15a76450bbc9197037e856e08 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:40.883752 5382 raft_consensus.cc:2932] T 1a7bbfc15a76450bbc9197037e856e08 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:40.889183 5365 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:40.891534 7154 raft_consensus.cc:2932] T 1a7bbfc15a76450bbc9197037e856e08 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:40.892096 6888 raft_consensus.cc:1261] T 2c39b53867e047faaf24cc6f87a8100c P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:05:40.892695 9391 consensus_queue.cc:1035] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:05:40.901722 4849 catalog_manager.cc:5484] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c reported cstate change: config changed from index 5 to 6, VOTER e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130) evicted. New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:40.912931 9382 tablet_copy_client.cc:783] T b284aa82594a44f5885f32fa05844913 P 4360b59e3a154688a79ca5f9c0a37bc9: tablet copy: Starting download of 0 data blocks...
I20240501 14:05:40.913163 9382 tablet_copy_client.cc:647] T b284aa82594a44f5885f32fa05844913 P 4360b59e3a154688a79ca5f9c0a37bc9: tablet copy: Starting download of 1 WAL segments...
I20240501 14:05:40.919242 9382 tablet_copy_client.cc:523] T b284aa82594a44f5885f32fa05844913 P 4360b59e3a154688a79ca5f9c0a37bc9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:05:40.919813 8375 raft_consensus.cc:1050] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410: attempting to promote NON_VOTER 938a24b0e6a040b7966225af26558ff8 to VOTER
I20240501 14:05:40.920682 8375 consensus_queue.cc:237] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:05:40.925051 4852 catalog_manager.cc:5484] T 1a7bbfc15a76450bbc9197037e856e08 P 19a6d2d8a08142d5bffcde3a37234692 reported cstate change: config changed from index 7 to 8, 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:40.931020 9390 raft_consensus.cc:2932] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } }
I20240501 14:05:40.932973 7267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:40.933619 9382 tablet_bootstrap.cc:492] T b284aa82594a44f5885f32fa05844913 P 4360b59e3a154688a79ca5f9c0a37bc9: Bootstrap starting.
I20240501 14:05:40.934090 5384 consensus_queue.cc:237] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } }
I20240501 14:05:40.936332 5126 raft_consensus.cc:1261] T 63650547ab444f45bb377bcb49f258ad P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240501 14:05:40.936698 5322 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 420642af520d45ffbc7f2d35f93e75a7. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:40.937001 9009 consensus_queue.cc:1035] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:40.937613 4849 catalog_manager.cc:5031] ChangeConfig:REMOVE_PEER RPC for tablet 23ca2bf1e59f44fcac8a48aff8d056b1 with cas_config_opid_index 8: aborting the task: latest config opid_index 9; task opid_index 8
I20240501 14:05:40.938275 5106 tablet_service.cc:1514] Processing DeleteTablet for tablet 420642af520d45ffbc7f2d35f93e75a7 with delete_type TABLET_DATA_TOMBSTONED (TS e9c60449cf3c46f8aca4ab6c535302a4 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41850
I20240501 14:05:40.938586 9577 tablet_replica.cc:330] T 420642af520d45ffbc7f2d35f93e75a7 P e9c60449cf3c46f8aca4ab6c535302a4: stopping tablet replica
I20240501 14:05:40.938779 9577 raft_consensus.cc:2229] T 420642af520d45ffbc7f2d35f93e75a7 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:05:40.938974 9577 raft_consensus.cc:2258] T 420642af520d45ffbc7f2d35f93e75a7 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:05:40.939646 9577 ts_tablet_manager.cc:1875] T 420642af520d45ffbc7f2d35f93e75a7 P e9c60449cf3c46f8aca4ab6c535302a4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:05:40.946810 9577 ts_tablet_manager.cc:1888] T 420642af520d45ffbc7f2d35f93e75a7 P e9c60449cf3c46f8aca4ab6c535302a4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:05:40.946810 7285 raft_consensus.cc:2932] T 2c39b53867e047faaf24cc6f87a8100c P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } }
I20240501 14:05:40.947645 9577 log.cc:1196] T 420642af520d45ffbc7f2d35f93e75a7 P e9c60449cf3c46f8aca4ab6c535302a4: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/wal/wals/420642af520d45ffbc7f2d35f93e75a7
I20240501 14:05:40.948877 4839 catalog_manager.cc:4830] TS e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377): tablet 420642af520d45ffbc7f2d35f93e75a7 (table rebalance_test_table_9 [id=5fc33bdfba464f65b68de5887d425ab7]) successfully deleted
I20240501 14:05:40.950443 5255 raft_consensus.cc:1261] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:05:40.952459 8804 consensus_queue.cc:1035] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:05:40.952562 6890 raft_consensus.cc:1261] T 63650547ab444f45bb377bcb49f258ad P 938a24b0e6a040b7966225af26558ff8 [term 1 LEARNER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:40.953420 8375 consensus_queue.cc:1035] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:40.954260 4847 catalog_manager.cc:5484] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf reported cstate change: config changed from index 11 to 12, NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) added. New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:05:40.956305 5255 raft_consensus.cc:1261] T 63650547ab444f45bb377bcb49f258ad P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:40.963229 5255 consensus_queue.cc:237] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } }
I20240501 14:05:40.965797 9569 consensus_queue.cc:1035] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:40.968725 7024 raft_consensus.cc:1261] T 1a7bbfc15a76450bbc9197037e856e08 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:05:40.970057 7025 raft_consensus.cc:1203] T f9cb784f60774ea7b6e70ccfebf867a8 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:05:40.972980 7156 raft_consensus.cc:1261] T 99d958617b7e4ef385b94f4c73f4dd84 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:40.973817 7154 raft_consensus.cc:1261] T 1a7bbfc15a76450bbc9197037e856e08 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:05:40.975005 8897 consensus_queue.cc:1035] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:05:40.975318 8375 consensus_queue.cc:1035] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:40.975688 8897 consensus_queue.cc:1035] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:05:40.978013 7136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:40.982309 7153 raft_consensus.cc:2932] T 99d958617b7e4ef385b94f4c73f4dd84 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:40.983695 9522 tablet_copy_client.cc:783] T a726769d24284fedadd8e9e63cd27d05 P 1c3fdc5272484ac09ef3e5e36a08f27c: tablet copy: Starting download of 0 data blocks...
I20240501 14:05:40.985226 9522 tablet_copy_client.cc:647] T a726769d24284fedadd8e9e63cd27d05 P 1c3fdc5272484ac09ef3e5e36a08f27c: tablet copy: Starting download of 1 WAL segments...
I20240501 14:05:40.984622 9576 raft_consensus.cc:2932] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER be56773105e744919601127a94e98adf (127.29.124.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:40.990352 9569 raft_consensus.cc:2932] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:40.991425 7154 raft_consensus.cc:2932] T 1a7bbfc15a76450bbc9197037e856e08 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER be56773105e744919601127a94e98adf (127.29.124.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:40.996733 5125 raft_consensus.cc:2932] T 63650547ab444f45bb377bcb49f258ad P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:40.999480 9522 tablet_copy_client.cc:523] T a726769d24284fedadd8e9e63cd27d05 P 1c3fdc5272484ac09ef3e5e36a08f27c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:05:41.003315 6890 raft_consensus.cc:2932] T 63650547ab444f45bb377bcb49f258ad P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:41.004055 4847 catalog_manager.cc:5484] T 63650547ab444f45bb377bcb49f258ad P e9c60449cf3c46f8aca4ab6c535302a4 reported cstate change: config changed from index 4 to 5, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "04088ee3bea440a7bd5ff772152b9410" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:41.008476 6892 raft_consensus.cc:1261] T 420642af520d45ffbc7f2d35f93e75a7 P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:05:41.013475 4854 catalog_manager.cc:5484] T 1a7bbfc15a76450bbc9197037e856e08 P 4360b59e3a154688a79ca5f9c0a37bc9 reported cstate change: config changed from index 8 to 9, VOTER be56773105e744919601127a94e98adf (127.29.124.132) evicted. New cstate: current_term: 1 leader_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:41.014456 7024 raft_consensus.cc:2932] T 1a7bbfc15a76450bbc9197037e856e08 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER be56773105e744919601127a94e98adf (127.29.124.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:41.023787 4838 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1a7bbfc15a76450bbc9197037e856e08 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:05:41.025099 6872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:41.027297 5255 raft_consensus.cc:2932] T 63650547ab444f45bb377bcb49f258ad P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:41.030048 9390 consensus_queue.cc:1035] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.021s
I20240501 14:05:41.036324 9522 tablet_bootstrap.cc:492] T a726769d24284fedadd8e9e63cd27d05 P 1c3fdc5272484ac09ef3e5e36a08f27c: Bootstrap starting.
I20240501 14:05:41.044754 9521 raft_consensus.cc:1050] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4: attempting to promote NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 to VOTER
I20240501 14:05:41.045639 9521 consensus_queue.cc:237] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } }
I20240501 14:05:41.048111 4993 consensus_queue.cc:237] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:05:41.051169 5255 raft_consensus.cc:1261] T 63650547ab444f45bb377bcb49f258ad P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:41.051182 6888 raft_consensus.cc:1261] T 63650547ab444f45bb377bcb49f258ad P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:41.051134 5254 raft_consensus.cc:2932] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:41.051640 9009 consensus_queue.cc:1035] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:41.052379 9569 consensus_queue.cc:1035] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:41.055016 9569 raft_consensus.cc:2932] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:41.055908 9390 raft_consensus.cc:2932] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:41.058141 9391 raft_consensus.cc:1050] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf: attempting to promote NON_VOTER 938a24b0e6a040b7966225af26558ff8 to VOTER
I20240501 14:05:41.058734 4839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 63650547ab444f45bb377bcb49f258ad with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:05:41.059006 9391 consensus_queue.cc:237] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } }
I20240501 14:05:41.059466 5254 raft_consensus.cc:2932] T 63650547ab444f45bb377bcb49f258ad P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:41.060739 9538 raft_consensus.cc:1050] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf: attempting to promote NON_VOTER 6dac90f01e744f9c8701a404440a4c77 to VOTER
I20240501 14:05:41.061475 9538 consensus_queue.cc:237] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } }
I20240501 14:05:41.064910 4852 catalog_manager.cc:5484] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 reported cstate change: config changed from index 5 to 6, VOTER e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130) evicted. New cstate: current_term: 1 leader_uuid: "04088ee3bea440a7bd5ff772152b9410" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:05:41.065778 4993 raft_consensus.cc:1261] T 88ed5438fc934adc9fb98c6cb4476598 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:41.066445 9051 consensus_queue.cc:1035] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:41.069773 5106 tablet_service.cc:1514] Processing DeleteTablet for tablet 63650547ab444f45bb377bcb49f258ad with delete_type TABLET_DATA_TOMBSTONED (TS e9c60449cf3c46f8aca4ab6c535302a4 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41850
I20240501 14:05:41.070207 9577 tablet_replica.cc:330] T 63650547ab444f45bb377bcb49f258ad P e9c60449cf3c46f8aca4ab6c535302a4: stopping tablet replica
I20240501 14:05:41.070413 9577 raft_consensus.cc:2229] T 63650547ab444f45bb377bcb49f258ad P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:05:41.070628 9577 raft_consensus.cc:2258] T 63650547ab444f45bb377bcb49f258ad P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:05:41.071437 9333 tablet_bootstrap.cc:492] T 3d6859d53fd747a5a5e94722626fa522 P 19a6d2d8a08142d5bffcde3a37234692: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:05:41.072736 9577 ts_tablet_manager.cc:1875] T 63650547ab444f45bb377bcb49f258ad P e9c60449cf3c46f8aca4ab6c535302a4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:05:41.079927 9577 ts_tablet_manager.cc:1888] T 63650547ab444f45bb377bcb49f258ad P e9c60449cf3c46f8aca4ab6c535302a4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:05:41.080133 9577 log.cc:1196] T 63650547ab444f45bb377bcb49f258ad P e9c60449cf3c46f8aca4ab6c535302a4: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/wal/wals/63650547ab444f45bb377bcb49f258ad
I20240501 14:05:41.080515 9333 tablet_bootstrap.cc:492] T 3d6859d53fd747a5a5e94722626fa522 P 19a6d2d8a08142d5bffcde3a37234692: Bootstrap complete.
I20240501 14:05:41.080835 9333 ts_tablet_manager.cc:1367] T 3d6859d53fd747a5a5e94722626fa522 P 19a6d2d8a08142d5bffcde3a37234692: Time spent bootstrapping tablet: real 1.394s user 0.053s sys 0.000s
I20240501 14:05:41.080946 4839 catalog_manager.cc:4830] TS e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377): tablet 63650547ab444f45bb377bcb49f258ad (table rebalance_test_table_7 [id=763c884aecd640b49c65c692fbde9422]) successfully deleted
I20240501 14:05:41.081832 7153 raft_consensus.cc:1261] T 88ed5438fc934adc9fb98c6cb4476598 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 LEARNER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:41.082049 9333 raft_consensus.cc:348] T 3d6859d53fd747a5a5e94722626fa522 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } }
I20240501 14:05:41.082350 9333 raft_consensus.cc:729] T 3d6859d53fd747a5a5e94722626fa522 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 19a6d2d8a08142d5bffcde3a37234692, State: Initialized, Role: LEARNER
I20240501 14:05:41.082455 9521 consensus_queue.cc:1035] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:41.082566 9333 consensus_queue.cc:260] T 3d6859d53fd747a5a5e94722626fa522 P 19a6d2d8a08142d5bffcde3a37234692 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } }
I20240501 14:05:41.083304 9333 ts_tablet_manager.cc:1398] T 3d6859d53fd747a5a5e94722626fa522 P 19a6d2d8a08142d5bffcde3a37234692: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:05:41.086817 5255 raft_consensus.cc:1261] T 88ed5438fc934adc9fb98c6cb4476598 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240501 14:05:41.088140 5146 tablet_copy_service.cc:342] P e9c60449cf3c46f8aca4ab6c535302a4: Request end of tablet copy session 19a6d2d8a08142d5bffcde3a37234692-3d6859d53fd747a5a5e94722626fa522 received from {username='slave'} at 127.29.124.134:41917
I20240501 14:05:41.088279 5146 tablet_copy_service.cc:433] P e9c60449cf3c46f8aca4ab6c535302a4: ending tablet copy session 19a6d2d8a08142d5bffcde3a37234692-3d6859d53fd747a5a5e94722626fa522 on tablet 3d6859d53fd747a5a5e94722626fa522 with peer 19a6d2d8a08142d5bffcde3a37234692
I20240501 14:05:41.089053 9521 consensus_queue.cc:1035] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240501 14:05:41.099016 6892 raft_consensus.cc:2932] T 63650547ab444f45bb377bcb49f258ad P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:41.109750 4993 raft_consensus.cc:2932] T 88ed5438fc934adc9fb98c6cb4476598 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
W20240501 14:05:41.111225 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:41.112461 6889 raft_consensus.cc:2932] T 420642af520d45ffbc7f2d35f93e75a7 P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:41.115208 4847 catalog_manager.cc:5484] T 88ed5438fc934adc9fb98c6cb4476598 P 04088ee3bea440a7bd5ff772152b9410 reported cstate change: config changed from index 4 to 5, 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:41.113940 9521 raft_consensus.cc:2932] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:41.122185 7153 raft_consensus.cc:2932] T 88ed5438fc934adc9fb98c6cb4476598 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:41.123126 5255 raft_consensus.cc:2932] T 88ed5438fc934adc9fb98c6cb4476598 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:41.127080 4853 catalog_manager.cc:5484] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c reported cstate change: config changed from index 6 to 7, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:41.129895 5126 consensus_queue.cc:237] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } }
I20240501 14:05:41.134661 5255 raft_consensus.cc:1261] T 88ed5438fc934adc9fb98c6cb4476598 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:41.135221 9051 consensus_queue.cc:1035] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:41.136247 7155 raft_consensus.cc:1261] T 88ed5438fc934adc9fb98c6cb4476598 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:41.136817 9051 consensus_queue.cc:1035] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:41.138809 9051 raft_consensus.cc:2932] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:41.144400 7153 raft_consensus.cc:2932] T 88ed5438fc934adc9fb98c6cb4476598 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:41.145665 4839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 88ed5438fc934adc9fb98c6cb4476598 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:05:41.146180 5255 raft_consensus.cc:2932] T 88ed5438fc934adc9fb98c6cb4476598 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:41.146585 4853 catalog_manager.cc:5484] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 reported cstate change: config changed from index 5 to 6, VOTER 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129) evicted. New cstate: current_term: 1 leader_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:05:41.154158 5106 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:41.158401 4973 tablet_service.cc:1514] Processing DeleteTablet for tablet 88ed5438fc934adc9fb98c6cb4476598 with delete_type TABLET_DATA_TOMBSTONED (TS 04088ee3bea440a7bd5ff772152b9410 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41194
I20240501 14:05:41.180712 9382 tablet_bootstrap.cc:492] T b284aa82594a44f5885f32fa05844913 P 4360b59e3a154688a79ca5f9c0a37bc9: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:05:41.180991 9619 tablet_replica.cc:330] T 88ed5438fc934adc9fb98c6cb4476598 P 04088ee3bea440a7bd5ff772152b9410: stopping tablet replica
I20240501 14:05:41.181208 9382 tablet_bootstrap.cc:492] T b284aa82594a44f5885f32fa05844913 P 4360b59e3a154688a79ca5f9c0a37bc9: Bootstrap complete.
I20240501 14:05:41.181344 9619 raft_consensus.cc:2229] T 88ed5438fc934adc9fb98c6cb4476598 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:05:41.181504 9382 ts_tablet_manager.cc:1367] T b284aa82594a44f5885f32fa05844913 P 4360b59e3a154688a79ca5f9c0a37bc9: Time spent bootstrapping tablet: real 0.248s user 0.046s sys 0.002s
I20240501 14:05:41.181820 9619 raft_consensus.cc:2258] T 88ed5438fc934adc9fb98c6cb4476598 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:05:41.182731 9619 ts_tablet_manager.cc:1875] T 88ed5438fc934adc9fb98c6cb4476598 P 04088ee3bea440a7bd5ff772152b9410: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:05:41.182679 9382 raft_consensus.cc:348] T b284aa82594a44f5885f32fa05844913 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } }
I20240501 14:05:41.182953 9382 raft_consensus.cc:729] T b284aa82594a44f5885f32fa05844913 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4360b59e3a154688a79ca5f9c0a37bc9, State: Initialized, Role: LEARNER
I20240501 14:05:41.183182 9382 consensus_queue.cc:260] T b284aa82594a44f5885f32fa05844913 P 4360b59e3a154688a79ca5f9c0a37bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } }
I20240501 14:05:41.187430 9382 ts_tablet_manager.cc:1398] T b284aa82594a44f5885f32fa05844913 P 4360b59e3a154688a79ca5f9c0a37bc9: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240501 14:05:41.187867 6889 raft_consensus.cc:2932] T 2c39b53867e047faaf24cc6f87a8100c P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } }
I20240501 14:05:41.190676 5406 tablet_copy_service.cc:342] P be56773105e744919601127a94e98adf: Request end of tablet copy session 4360b59e3a154688a79ca5f9c0a37bc9-b284aa82594a44f5885f32fa05844913 received from {username='slave'} at 127.29.124.135:51309
I20240501 14:05:41.190753 6892 raft_consensus.cc:1261] T 9b88c43cb8704583a3e65c5c35c205f6 P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:05:41.190826 5406 tablet_copy_service.cc:433] P be56773105e744919601127a94e98adf: ending tablet copy session 4360b59e3a154688a79ca5f9c0a37bc9-b284aa82594a44f5885f32fa05844913 on tablet b284aa82594a44f5885f32fa05844913 with peer 4360b59e3a154688a79ca5f9c0a37bc9
I20240501 14:05:41.191305 8804 consensus_queue.cc:1035] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:05:41.193500 6890 raft_consensus.cc:1261] T b284aa82594a44f5885f32fa05844913 P 938a24b0e6a040b7966225af26558ff8 [term 1 LEARNER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:41.194325 9391 consensus_queue.cc:1035] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:41.194949 5126 raft_consensus.cc:1261] T b284aa82594a44f5885f32fa05844913 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:41.195446 4993 raft_consensus.cc:1261] T b284aa82594a44f5885f32fa05844913 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:41.195531 9616 consensus_queue.cc:1035] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:41.196476 9390 consensus_queue.cc:1035] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:41.211097 9619 ts_tablet_manager.cc:1888] T 88ed5438fc934adc9fb98c6cb4476598 P 04088ee3bea440a7bd5ff772152b9410: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:05:41.212430 5255 raft_consensus.cc:1261] T 9b88c43cb8704583a3e65c5c35c205f6 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:05:41.213207 9538 consensus_queue.cc:1035] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:05:41.221292 7285 raft_consensus.cc:1261] T 9b88c43cb8704583a3e65c5c35c205f6 P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:05:41.221366 5365 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a7bbfc15a76450bbc9197037e856e08 with delete_type TABLET_DATA_TOMBSTONED (TS be56773105e744919601127a94e98adf not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:43020
I20240501 14:05:41.221912 9550 tablet_replica.cc:330] T 1a7bbfc15a76450bbc9197037e856e08 P be56773105e744919601127a94e98adf: stopping tablet replica
I20240501 14:05:41.222138 9550 raft_consensus.cc:2229] T 1a7bbfc15a76450bbc9197037e856e08 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:05:41.223304 9550 raft_consensus.cc:2258] T 1a7bbfc15a76450bbc9197037e856e08 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:05:41.224517 9550 ts_tablet_manager.cc:1875] T 1a7bbfc15a76450bbc9197037e856e08 P be56773105e744919601127a94e98adf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:05:41.225126 9538 consensus_queue.cc:1035] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:05:41.232357 9550 ts_tablet_manager.cc:1888] T 1a7bbfc15a76450bbc9197037e856e08 P be56773105e744919601127a94e98adf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 14:05:41.232553 9550 log.cc:1196] T 1a7bbfc15a76450bbc9197037e856e08 P be56773105e744919601127a94e98adf: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/wal/wals/1a7bbfc15a76450bbc9197037e856e08
I20240501 14:05:41.240578 9538 raft_consensus.cc:2932] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:41.243314 9618 ts_tablet_manager.cc:902] T 8cc5eec4600640f7a293877c825f8d96 P 938a24b0e6a040b7966225af26558ff8: Initiating tablet copy from peer 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129:44033)
I20240501 14:05:41.246294 5125 raft_consensus.cc:2932] T b284aa82594a44f5885f32fa05844913 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:41.249176 4839 catalog_manager.cc:4830] TS be56773105e744919601127a94e98adf (127.29.124.132:35459): tablet 1a7bbfc15a76450bbc9197037e856e08 (table rebalance_test_table_1 [id=403fdb578f8e4066a56f685925e83936]) successfully deleted
I20240501 14:05:41.239897 4992 raft_consensus.cc:2932] T b284aa82594a44f5885f32fa05844913 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:41.255067 4847 catalog_manager.cc:5484] T b284aa82594a44f5885f32fa05844913 P e9c60449cf3c46f8aca4ab6c535302a4 reported cstate change: config changed from index 5 to 6, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:41.257889 9619 log.cc:1196] T 88ed5438fc934adc9fb98c6cb4476598 P 04088ee3bea440a7bd5ff772152b9410: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/wal/wals/88ed5438fc934adc9fb98c6cb4476598
I20240501 14:05:41.259155 4839 catalog_manager.cc:4830] TS 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129:44033): tablet 88ed5438fc934adc9fb98c6cb4476598 (table rebalance_test_table_0 [id=7af67fe405db4bffbc53cb288b8bb715]) successfully deleted
I20240501 14:05:41.261224 5382 consensus_queue.cc:237] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } }
I20240501 14:05:41.263893 7153 raft_consensus.cc:1261] T f4d2b8c005b64bbb8e6621da068d15e5 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:05:41.265100 7024 raft_consensus.cc:1261] T f4d2b8c005b64bbb8e6621da068d15e5 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:05:41.265172 9618 tablet_copy_client.cc:308] T 8cc5eec4600640f7a293877c825f8d96 P 938a24b0e6a040b7966225af26558ff8: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.129:44033
I20240501 14:05:41.265839 5015 tablet_copy_service.cc:140] P 04088ee3bea440a7bd5ff772152b9410: Received BeginTabletCopySession request for tablet 8cc5eec4600640f7a293877c825f8d96 from peer 938a24b0e6a040b7966225af26558ff8 ({username='slave'} at 127.29.124.133:49901)
I20240501 14:05:41.266016 5015 tablet_copy_service.cc:161] P 04088ee3bea440a7bd5ff772152b9410: Beginning new tablet copy session on tablet 8cc5eec4600640f7a293877c825f8d96 from peer 938a24b0e6a040b7966225af26558ff8 at {username='slave'} at 127.29.124.133:49901: session id = 938a24b0e6a040b7966225af26558ff8-8cc5eec4600640f7a293877c825f8d96
I20240501 14:05:41.268858 5015 tablet_copy_source_session.cc:205] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:05:41.269876 8891 consensus_queue.cc:1035] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:05:41.271221 8804 consensus_queue.cc:1035] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:05:41.272022 9616 raft_consensus.cc:2932] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:41.274996 8804 raft_consensus.cc:2932] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:41.276796 6888 raft_consensus.cc:2932] T b284aa82594a44f5885f32fa05844913 P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:41.279028 4847 catalog_manager.cc:5484] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf reported cstate change: config changed from index 9 to 10, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:05:41.281886 9618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8cc5eec4600640f7a293877c825f8d96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:41.280658 6892 raft_consensus.cc:2932] T 9b88c43cb8704583a3e65c5c35c205f6 P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:41.285068 7024 raft_consensus.cc:2932] T f4d2b8c005b64bbb8e6621da068d15e5 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:41.285820 4839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f4d2b8c005b64bbb8e6621da068d15e5 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:05:41.290697 7153 raft_consensus.cc:2932] T f4d2b8c005b64bbb8e6621da068d15e5 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:41.290870 4847 catalog_manager.cc:5484] T f4d2b8c005b64bbb8e6621da068d15e5 P 19a6d2d8a08142d5bffcde3a37234692 reported cstate change: config changed from index 10 to 11, VOTER 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) evicted. New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } }
I20240501 14:05:41.294224 9618 tablet_copy_client.cc:783] T 8cc5eec4600640f7a293877c825f8d96 P 938a24b0e6a040b7966225af26558ff8: tablet copy: Starting download of 0 data blocks...
I20240501 14:05:41.295276 5236 tablet_service.cc:1514] Processing DeleteTablet for tablet f4d2b8c005b64bbb8e6621da068d15e5 with delete_type TABLET_DATA_TOMBSTONED (TS 1c3fdc5272484ac09ef3e5e36a08f27c not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:39020
I20240501 14:05:41.295169 5255 raft_consensus.cc:2932] T 9b88c43cb8704583a3e65c5c35c205f6 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:41.295696 9508 tablet_replica.cc:330] T f4d2b8c005b64bbb8e6621da068d15e5 P 1c3fdc5272484ac09ef3e5e36a08f27c: stopping tablet replica
I20240501 14:05:41.295173 9618 tablet_copy_client.cc:647] T 8cc5eec4600640f7a293877c825f8d96 P 938a24b0e6a040b7966225af26558ff8: tablet copy: Starting download of 1 WAL segments...
I20240501 14:05:41.295903 9508 raft_consensus.cc:2229] T f4d2b8c005b64bbb8e6621da068d15e5 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:05:41.296340 9508 raft_consensus.cc:2258] T f4d2b8c005b64bbb8e6621da068d15e5 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:05:41.297539 9508 ts_tablet_manager.cc:1875] T f4d2b8c005b64bbb8e6621da068d15e5 P 1c3fdc5272484ac09ef3e5e36a08f27c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:05:41.299243 9618 tablet_copy_client.cc:523] T 8cc5eec4600640f7a293877c825f8d96 P 938a24b0e6a040b7966225af26558ff8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:05:41.303498 9508 ts_tablet_manager.cc:1888] T f4d2b8c005b64bbb8e6621da068d15e5 P 1c3fdc5272484ac09ef3e5e36a08f27c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240501 14:05:41.303670 9508 log.cc:1196] T f4d2b8c005b64bbb8e6621da068d15e5 P 1c3fdc5272484ac09ef3e5e36a08f27c: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/wal/wals/f4d2b8c005b64bbb8e6621da068d15e5
I20240501 14:05:41.304636 4838 catalog_manager.cc:4830] TS 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209): tablet f4d2b8c005b64bbb8e6621da068d15e5 (table rebalance_test_table_1 [id=403fdb578f8e4066a56f685925e83936]) successfully deleted
I20240501 14:05:41.307188 5382 consensus_queue.cc:237] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:05:41.311837 5126 raft_consensus.cc:1261] T b284aa82594a44f5885f32fa05844913 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:05:41.312443 4993 raft_consensus.cc:1261] T b284aa82594a44f5885f32fa05844913 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:05:41.313320 9390 consensus_queue.cc:1035] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:05:41.313788 6892 raft_consensus.cc:1261] T b284aa82594a44f5885f32fa05844913 P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:05:41.313956 9618 tablet_bootstrap.cc:492] T 8cc5eec4600640f7a293877c825f8d96 P 938a24b0e6a040b7966225af26558ff8: Bootstrap starting.
I20240501 14:05:41.314369 8891 consensus_queue.cc:1035] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:05:41.314893 8891 consensus_queue.cc:1035] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:05:41.323344 9608 raft_consensus.cc:2932] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:41.323963 6888 raft_consensus.cc:2932] T b284aa82594a44f5885f32fa05844913 P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:41.328707 4839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b284aa82594a44f5885f32fa05844913 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:05:41.330379 4993 raft_consensus.cc:2932] T b284aa82594a44f5885f32fa05844913 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:41.330397 5126 raft_consensus.cc:2932] T b284aa82594a44f5885f32fa05844913 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:41.335616 4847 catalog_manager.cc:5484] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf reported cstate change: config changed from index 6 to 7, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) evicted. New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:05:41.339967 7136 tablet_service.cc:1514] Processing DeleteTablet for tablet b284aa82594a44f5885f32fa05844913 with delete_type TABLET_DATA_TOMBSTONED (TS 4360b59e3a154688a79ca5f9c0a37bc9 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:51170
I20240501 14:05:41.355731 7004 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:41.364822 9641 tablet_replica.cc:330] T b284aa82594a44f5885f32fa05844913 P 4360b59e3a154688a79ca5f9c0a37bc9: stopping tablet replica
I20240501 14:05:41.369643 9641 raft_consensus.cc:2229] T b284aa82594a44f5885f32fa05844913 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 LEARNER]: Raft consensus shutting down.
I20240501 14:05:41.371228 9641 raft_consensus.cc:2258] T b284aa82594a44f5885f32fa05844913 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 LEARNER]: Raft consensus is shut down!
I20240501 14:05:41.374094 9641 ts_tablet_manager.cc:1875] T b284aa82594a44f5885f32fa05844913 P 4360b59e3a154688a79ca5f9c0a37bc9: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:05:41.380174 7285 raft_consensus.cc:2932] T 9b88c43cb8704583a3e65c5c35c205f6 P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:41.383121 5382 consensus_queue.cc:237] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } }
I20240501 14:05:41.385190 6892 raft_consensus.cc:1261] T 9b88c43cb8704583a3e65c5c35c205f6 P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:05:41.385834 9616 consensus_queue.cc:1035] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:05:41.388382 9616 raft_consensus.cc:2932] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:41.392051 7286 raft_consensus.cc:1261] T 9b88c43cb8704583a3e65c5c35c205f6 P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:05:41.392330 4839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9b88c43cb8704583a3e65c5c35c205f6 with cas_config_opid_index 10: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:05:41.392995 6888 raft_consensus.cc:2932] T 9b88c43cb8704583a3e65c5c35c205f6 P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:41.393563 9390 consensus_queue.cc:1035] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:05:41.396024 4854 catalog_manager.cc:5484] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf reported cstate change: config changed from index 10 to 11, VOTER 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) evicted. New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:05:41.396241 7285 raft_consensus.cc:2932] T 9b88c43cb8704583a3e65c5c35c205f6 P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, VOTER 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) evicted. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:41.403173 5236 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b88c43cb8704583a3e65c5c35c205f6 with delete_type TABLET_DATA_TOMBSTONED (TS 1c3fdc5272484ac09ef3e5e36a08f27c not found in new config with opid_index 11) from {username='slave'} at 127.0.0.1:39020
I20240501 14:05:41.403540 9508 tablet_replica.cc:330] T 9b88c43cb8704583a3e65c5c35c205f6 P 1c3fdc5272484ac09ef3e5e36a08f27c: stopping tablet replica
I20240501 14:05:41.403766 9508 raft_consensus.cc:2229] T 9b88c43cb8704583a3e65c5c35c205f6 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:05:41.404013 9508 raft_consensus.cc:2258] T 9b88c43cb8704583a3e65c5c35c205f6 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:05:41.405021 9508 ts_tablet_manager.cc:1875] T 9b88c43cb8704583a3e65c5c35c205f6 P 1c3fdc5272484ac09ef3e5e36a08f27c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:05:41.415011 4973 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:41.421355 9009 raft_consensus.cc:1050] T a726769d24284fedadd8e9e63cd27d05 P 04088ee3bea440a7bd5ff772152b9410: attempting to promote NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 to VOTER
I20240501 14:05:41.422235 9009 consensus_queue.cc:237] T a726769d24284fedadd8e9e63cd27d05 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: NON_VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { promote: true } }
I20240501 14:05:41.431223 7024 raft_consensus.cc:1261] T a726769d24284fedadd8e9e63cd27d05 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 LEARNER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:41.433699 9569 consensus_queue.cc:1035] T a726769d24284fedadd8e9e63cd27d05 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240501 14:05:41.436278 4928 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209): Couldn't send request to peer 1c3fdc5272484ac09ef3e5e36a08f27c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:41.442308 9520 raft_consensus.cc:1050] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4: attempting to promote NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 to VOTER
I20240501 14:05:41.443153 9520 consensus_queue.cc:237] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } }
I20240501 14:05:41.445668 5126 raft_consensus.cc:1261] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:41.447350 9333 ts_tablet_manager.cc:902] T 2c39b53867e047faaf24cc6f87a8100c P 19a6d2d8a08142d5bffcde3a37234692: Initiating tablet copy from peer be56773105e744919601127a94e98adf (127.29.124.132:35459)
I20240501 14:05:41.448177 5382 raft_consensus.cc:1261] T a726769d24284fedadd8e9e63cd27d05 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240501 14:05:41.448706 9569 consensus_queue.cc:1035] T a726769d24284fedadd8e9e63cd27d05 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:41.449210 9569 consensus_queue.cc:1035] T a726769d24284fedadd8e9e63cd27d05 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240501 14:05:41.454964 4993 consensus_queue.cc:237] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } }
W20240501 14:05:41.458271 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20240501 14:05:41.458961 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8cc5eec4600640f7a293877c825f8d96. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:41.459326 7025 raft_consensus.cc:1261] T 8cc5eec4600640f7a293877c825f8d96 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:05:41.459914 9569 consensus_queue.cc:1035] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:05:41.460093 5125 raft_consensus.cc:1261] T 8cc5eec4600640f7a293877c825f8d96 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:05:41.461478 9569 consensus_queue.cc:1035] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:05:41.462102 8375 raft_consensus.cc:2932] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:41.462740 7025 raft_consensus.cc:2932] T 8cc5eec4600640f7a293877c825f8d96 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:41.464329 5125 raft_consensus.cc:2932] T 8cc5eec4600640f7a293877c825f8d96 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:41.468869 4852 catalog_manager.cc:5484] T 8cc5eec4600640f7a293877c825f8d96 P 19a6d2d8a08142d5bffcde3a37234692 reported cstate change: config changed from index 9 to 10, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) added. New cstate: current_term: 1 leader_uuid: "04088ee3bea440a7bd5ff772152b9410" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:41.474272 7136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:41.477159 9569 raft_consensus.cc:2932] T a726769d24284fedadd8e9e63cd27d05 P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: NON_VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { promote: true } } }
I20240501 14:05:41.482156 7024 raft_consensus.cc:2932] T a726769d24284fedadd8e9e63cd27d05 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: NON_VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { promote: true } } }
I20240501 14:05:41.501961 5125 raft_consensus.cc:2932] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: NON_VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { promote: true } } }
I20240501 14:05:41.505851 9618 tablet_bootstrap.cc:492] T 8cc5eec4600640f7a293877c825f8d96 P 938a24b0e6a040b7966225af26558ff8: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:05:41.508605 9618 tablet_bootstrap.cc:492] T 8cc5eec4600640f7a293877c825f8d96 P 938a24b0e6a040b7966225af26558ff8: Bootstrap complete.
I20240501 14:05:41.508949 9618 ts_tablet_manager.cc:1367] T 8cc5eec4600640f7a293877c825f8d96 P 938a24b0e6a040b7966225af26558ff8: Time spent bootstrapping tablet: real 0.195s user 0.049s sys 0.007s
I20240501 14:05:41.508795 4853 catalog_manager.cc:5484] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 reported cstate change: config changed from index 5 to 6, 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "04088ee3bea440a7bd5ff772152b9410" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: NON_VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { promote: true } } }
I20240501 14:05:41.509817 9618 raft_consensus.cc:348] T 8cc5eec4600640f7a293877c825f8d96 P 938a24b0e6a040b7966225af26558ff8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }
I20240501 14:05:41.510110 9618 raft_consensus.cc:729] T 8cc5eec4600640f7a293877c825f8d96 P 938a24b0e6a040b7966225af26558ff8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 938a24b0e6a040b7966225af26558ff8, State: Initialized, Role: LEARNER
I20240501 14:05:41.510308 9618 consensus_queue.cc:260] T 8cc5eec4600640f7a293877c825f8d96 P 938a24b0e6a040b7966225af26558ff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }
I20240501 14:05:41.510929 9618 ts_tablet_manager.cc:1398] T 8cc5eec4600640f7a293877c825f8d96 P 938a24b0e6a040b7966225af26558ff8: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:05:41.519840 5015 tablet_copy_service.cc:342] P 04088ee3bea440a7bd5ff772152b9410: Request end of tablet copy session 938a24b0e6a040b7966225af26558ff8-8cc5eec4600640f7a293877c825f8d96 received from {username='slave'} at 127.29.124.133:49901
I20240501 14:05:41.520027 5015 tablet_copy_service.cc:433] P 04088ee3bea440a7bd5ff772152b9410: ending tablet copy session 938a24b0e6a040b7966225af26558ff8-8cc5eec4600640f7a293877c825f8d96 on tablet 8cc5eec4600640f7a293877c825f8d96 with peer 938a24b0e6a040b7966225af26558ff8
I20240501 14:05:41.520027 9508 ts_tablet_manager.cc:1888] T 9b88c43cb8704583a3e65c5c35c205f6 P 1c3fdc5272484ac09ef3e5e36a08f27c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.10
I20240501 14:05:41.520181 9508 log.cc:1196] T 9b88c43cb8704583a3e65c5c35c205f6 P 1c3fdc5272484ac09ef3e5e36a08f27c: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/wal/wals/9b88c43cb8704583a3e65c5c35c205f6
I20240501 14:05:41.521109 4838 catalog_manager.cc:4830] TS 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209): tablet 9b88c43cb8704583a3e65c5c35c205f6 (table rebalance_test_table_0 [id=7af67fe405db4bffbc53cb288b8bb715]) successfully deleted
I20240501 14:05:41.539234 5236 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:41.546152 9659 ts_tablet_manager.cc:902] T 420642af520d45ffbc7f2d35f93e75a7 P 6dac90f01e744f9c8701a404440a4c77: Initiating tablet copy from peer be56773105e744919601127a94e98adf (127.29.124.132:35459)
I20240501 14:05:41.570554 7024 raft_consensus.cc:1203] T 3d6859d53fd747a5a5e94722626fa522 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:05:41.571293 9333 tablet_copy_client.cc:308] T 2c39b53867e047faaf24cc6f87a8100c P 19a6d2d8a08142d5bffcde3a37234692: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.132:35459
I20240501 14:05:41.572755 5406 tablet_copy_service.cc:140] P be56773105e744919601127a94e98adf: Received BeginTabletCopySession request for tablet 2c39b53867e047faaf24cc6f87a8100c from peer 19a6d2d8a08142d5bffcde3a37234692 ({username='slave'} at 127.29.124.134:56125)
I20240501 14:05:41.572943 5406 tablet_copy_service.cc:161] P be56773105e744919601127a94e98adf: Beginning new tablet copy session on tablet 2c39b53867e047faaf24cc6f87a8100c from peer 19a6d2d8a08142d5bffcde3a37234692 at {username='slave'} at 127.29.124.134:56125: session id = 19a6d2d8a08142d5bffcde3a37234692-2c39b53867e047faaf24cc6f87a8100c
I20240501 14:05:41.576049 5406 tablet_copy_source_session.cc:205] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:05:41.577436 9333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c39b53867e047faaf24cc6f87a8100c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:41.587384 5386 raft_consensus.cc:1261] T f9cb784f60774ea7b6e70ccfebf867a8 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:05:41.587951 9051 consensus_queue.cc:1035] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:05:41.595738 7024 raft_consensus.cc:1261] T f9cb784f60774ea7b6e70ccfebf867a8 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 LEARNER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:05:41.599027 9051 consensus_queue.cc:1035] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:05:41.603534 7285 raft_consensus.cc:1261] T f9cb784f60774ea7b6e70ccfebf867a8 P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:05:41.604384 9051 consensus_queue.cc:1035] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:05:41.608326 9333 tablet_copy_client.cc:783] T 2c39b53867e047faaf24cc6f87a8100c P 19a6d2d8a08142d5bffcde3a37234692: tablet copy: Starting download of 0 data blocks...
I20240501 14:05:41.608601 9333 tablet_copy_client.cc:647] T 2c39b53867e047faaf24cc6f87a8100c P 19a6d2d8a08142d5bffcde3a37234692: tablet copy: Starting download of 1 WAL segments...
I20240501 14:05:41.614480 9333 tablet_copy_client.cc:523] T 2c39b53867e047faaf24cc6f87a8100c P 19a6d2d8a08142d5bffcde3a37234692: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:05:41.618242 9333 tablet_bootstrap.cc:492] T 2c39b53867e047faaf24cc6f87a8100c P 19a6d2d8a08142d5bffcde3a37234692: Bootstrap starting.
I20240501 14:05:41.626111 5125 consensus_queue.cc:237] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } }
I20240501 14:05:41.629976 4993 consensus_queue.cc:237] T a726769d24284fedadd8e9e63cd27d05 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } }
I20240501 14:05:41.630416 5382 raft_consensus.cc:2932] T a726769d24284fedadd8e9e63cd27d05 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: NON_VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { promote: true } } }
I20240501 14:05:41.633018 7025 raft_consensus.cc:1261] T a726769d24284fedadd8e9e63cd27d05 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:05:41.634203 5126 raft_consensus.cc:1261] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:05:41.634418 9569 consensus_queue.cc:1035] T a726769d24284fedadd8e9e63cd27d05 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:05:41.636054 9569 consensus_queue.cc:1035] T a726769d24284fedadd8e9e63cd27d05 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:05:41.638530 9569 raft_consensus.cc:2932] T a726769d24284fedadd8e9e63cd27d05 P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:41.642817 4839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a726769d24284fedadd8e9e63cd27d05 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:05:41.643244 7025 raft_consensus.cc:2932] T a726769d24284fedadd8e9e63cd27d05 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:41.643654 5126 raft_consensus.cc:2932] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:41.650013 4853 catalog_manager.cc:5484] T a726769d24284fedadd8e9e63cd27d05 P 04088ee3bea440a7bd5ff772152b9410 reported cstate change: config changed from index 6 to 7, NON_VOTER 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) evicted. New cstate: current_term: 1 leader_uuid: "04088ee3bea440a7bd5ff772152b9410" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:05:41.672767 7285 raft_consensus.cc:1261] T 9d32e0707d244f2c9bbcb57f62b09f31 P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 14:05:41.672853 5061 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9d32e0707d244f2c9bbcb57f62b09f31. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:41.673432 9520 consensus_queue.cc:1035] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:05:41.681663 5236 tablet_service.cc:1514] Processing DeleteTablet for tablet a726769d24284fedadd8e9e63cd27d05 with delete_type TABLET_DATA_TOMBSTONED (TS 1c3fdc5272484ac09ef3e5e36a08f27c not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:39020
I20240501 14:05:41.683534 5385 raft_consensus.cc:1261] T 9d32e0707d244f2c9bbcb57f62b09f31 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 14:05:41.683809 4838 catalog_manager.cc:4810] TS 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209): delete failed for tablet a726769d24284fedadd8e9e63cd27d05 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a726769d24284fedadd8e9e63cd27d05 already in progress: copying tablet
I20240501 14:05:41.684702 9520 consensus_queue.cc:1035] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:05:41.690984 9520 raft_consensus.cc:2932] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:41.697211 7285 raft_consensus.cc:2932] T 9d32e0707d244f2c9bbcb57f62b09f31 P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:41.713111 9641 ts_tablet_manager.cc:1888] T b284aa82594a44f5885f32fa05844913 P 4360b59e3a154688a79ca5f9c0a37bc9: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:05:41.713363 9641 log.cc:1196] T b284aa82594a44f5885f32fa05844913 P 4360b59e3a154688a79ca5f9c0a37bc9: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-6/wal/wals/b284aa82594a44f5885f32fa05844913
I20240501 14:05:41.714282 4840 catalog_manager.cc:4830] TS 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): tablet b284aa82594a44f5885f32fa05844913 (table rebalance_test_table_4 [id=84aeba643ddd44cfbbd6a34f81703482]) successfully deleted
I20240501 14:05:41.723614 5386 consensus_queue.cc:237] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }
I20240501 14:05:41.727535 9490 raft_consensus.cc:2932] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:41.728190 7286 raft_consensus.cc:2932] T f9cb784f60774ea7b6e70ccfebf867a8 P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:41.740741 7024 raft_consensus.cc:2932] T f9cb784f60774ea7b6e70ccfebf867a8 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:41.741544 7267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:41.740170 4853 catalog_manager.cc:5484] T 9d32e0707d244f2c9bbcb57f62b09f31 P 6dac90f01e744f9c8701a404440a4c77 reported cstate change: config changed from index 6 to 7, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) added. New cstate: current_term: 1 leader_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:41.756433 4854 catalog_manager.cc:5484] T f9cb784f60774ea7b6e70ccfebf867a8 P 19a6d2d8a08142d5bffcde3a37234692 reported cstate change: config changed from index 7 to 8, 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:41.764456 5106 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:41.769790 5384 raft_consensus.cc:1261] T a726769d24284fedadd8e9e63cd27d05 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:05:41.770180 5383 raft_consensus.cc:2932] T f9cb784f60774ea7b6e70ccfebf867a8 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:41.772293 5385 raft_consensus.cc:2932] T 9d32e0707d244f2c9bbcb57f62b09f31 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:41.772578 8375 consensus_queue.cc:1035] T a726769d24284fedadd8e9e63cd27d05 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:05:41.779767 5384 raft_consensus.cc:2932] T a726769d24284fedadd8e9e63cd27d05 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:41.783191 6872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
W20240501 14:05:41.809123 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:41.810146 7286 raft_consensus.cc:1261] T e7d5f02ccba34254a4d20951878b1214 P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:05:41.810348 5365 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:41.810708 9390 consensus_queue.cc:1035] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:05:41.810799 7024 raft_consensus.cc:1261] T e7d5f02ccba34254a4d20951878b1214 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:05:41.811321 9390 consensus_queue.cc:1035] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:05:41.813709 9659 tablet_copy_client.cc:308] T 420642af520d45ffbc7f2d35f93e75a7 P 6dac90f01e744f9c8701a404440a4c77: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.132:35459
I20240501 14:05:41.814404 5406 tablet_copy_service.cc:140] P be56773105e744919601127a94e98adf: Received BeginTabletCopySession request for tablet 420642af520d45ffbc7f2d35f93e75a7 from peer 6dac90f01e744f9c8701a404440a4c77 ({username='slave'} at 127.29.124.136:50719)
I20240501 14:05:41.814554 5406 tablet_copy_service.cc:161] P be56773105e744919601127a94e98adf: Beginning new tablet copy session on tablet 420642af520d45ffbc7f2d35f93e75a7 from peer 6dac90f01e744f9c8701a404440a4c77 at {username='slave'} at 127.29.124.136:50719: session id = 6dac90f01e744f9c8701a404440a4c77-420642af520d45ffbc7f2d35f93e75a7
I20240501 14:05:41.817742 5406 tablet_copy_source_session.cc:205] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:05:41.822980 5125 consensus_queue.cc:237] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } }
I20240501 14:05:41.825176 7025 raft_consensus.cc:1261] T f9cb784f60774ea7b6e70ccfebf867a8 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:05:41.826049 9688 consensus_queue.cc:1035] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:05:41.826205 7285 raft_consensus.cc:1261] T f9cb784f60774ea7b6e70ccfebf867a8 P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:05:41.826712 9688 consensus_queue.cc:1035] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:05:41.830067 9688 raft_consensus.cc:2932] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER be56773105e744919601127a94e98adf (127.29.124.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:41.831439 7285 raft_consensus.cc:2932] T f9cb784f60774ea7b6e70ccfebf867a8 P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER be56773105e744919601127a94e98adf (127.29.124.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:41.834702 4839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f9cb784f60774ea7b6e70ccfebf867a8 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:05:41.835462 9659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 420642af520d45ffbc7f2d35f93e75a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:41.835445 7025 raft_consensus.cc:2932] T f9cb784f60774ea7b6e70ccfebf867a8 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER be56773105e744919601127a94e98adf (127.29.124.132) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:41.837566 8891 raft_consensus.cc:2932] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:41.838141 7287 raft_consensus.cc:2932] T e7d5f02ccba34254a4d20951878b1214 P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:41.840778 7023 raft_consensus.cc:2932] T e7d5f02ccba34254a4d20951878b1214 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:41.851714 4854 catalog_manager.cc:5484] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 reported cstate change: config changed from index 8 to 9, VOTER be56773105e744919601127a94e98adf (127.29.124.132) evicted. New cstate: current_term: 1 leader_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:05:41.853374 4847 catalog_manager.cc:5484] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf reported cstate change: config changed from index 12 to 13, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:05:41.857403 5365 tablet_service.cc:1514] Processing DeleteTablet for tablet f9cb784f60774ea7b6e70ccfebf867a8 with delete_type TABLET_DATA_TOMBSTONED (TS be56773105e744919601127a94e98adf not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:43020
I20240501 14:05:41.860337 9522 tablet_bootstrap.cc:492] T a726769d24284fedadd8e9e63cd27d05 P 1c3fdc5272484ac09ef3e5e36a08f27c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:05:41.860831 9522 tablet_bootstrap.cc:492] T a726769d24284fedadd8e9e63cd27d05 P 1c3fdc5272484ac09ef3e5e36a08f27c: Bootstrap complete.
I20240501 14:05:41.861137 9522 ts_tablet_manager.cc:1367] T a726769d24284fedadd8e9e63cd27d05 P 1c3fdc5272484ac09ef3e5e36a08f27c: Time spent bootstrapping tablet: real 0.825s user 0.071s sys 0.022s
I20240501 14:05:41.864297 9522 raft_consensus.cc:348] T a726769d24284fedadd8e9e63cd27d05 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: NON_VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { promote: true } }
I20240501 14:05:41.865522 9522 raft_consensus.cc:729] T a726769d24284fedadd8e9e63cd27d05 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1c3fdc5272484ac09ef3e5e36a08f27c, State: Initialized, Role: LEARNER
I20240501 14:05:41.865746 9522 consensus_queue.cc:260] T a726769d24284fedadd8e9e63cd27d05 P 1c3fdc5272484ac09ef3e5e36a08f27c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: NON_VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { promote: true } }
I20240501 14:05:41.866345 9522 ts_tablet_manager.cc:1398] T a726769d24284fedadd8e9e63cd27d05 P 1c3fdc5272484ac09ef3e5e36a08f27c: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20240501 14:05:41.869004 5236 tablet_service.cc:1514] Processing DeleteTablet for tablet a726769d24284fedadd8e9e63cd27d05 with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 5 (current committed config index is 7)) from {username='slave'} at 127.0.0.1:39020
I20240501 14:05:41.870077 9508 tablet_replica.cc:330] T a726769d24284fedadd8e9e63cd27d05 P 1c3fdc5272484ac09ef3e5e36a08f27c: stopping tablet replica
I20240501 14:05:41.870275 9508 raft_consensus.cc:2229] T a726769d24284fedadd8e9e63cd27d05 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEARNER]: Raft consensus shutting down.
I20240501 14:05:41.870443 9508 raft_consensus.cc:2258] T a726769d24284fedadd8e9e63cd27d05 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEARNER]: Raft consensus is shut down!
I20240501 14:05:41.870995 9508 ts_tablet_manager.cc:1875] T a726769d24284fedadd8e9e63cd27d05 P 1c3fdc5272484ac09ef3e5e36a08f27c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:05:41.875830 9659 tablet_copy_client.cc:783] T 420642af520d45ffbc7f2d35f93e75a7 P 6dac90f01e744f9c8701a404440a4c77: tablet copy: Starting download of 0 data blocks...
I20240501 14:05:41.876097 9659 tablet_copy_client.cc:647] T 420642af520d45ffbc7f2d35f93e75a7 P 6dac90f01e744f9c8701a404440a4c77: tablet copy: Starting download of 1 WAL segments...
I20240501 14:05:41.879101 9659 tablet_copy_client.cc:523] T 420642af520d45ffbc7f2d35f93e75a7 P 6dac90f01e744f9c8701a404440a4c77: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:05:41.882055 9508 ts_tablet_manager.cc:1888] T a726769d24284fedadd8e9e63cd27d05 P 1c3fdc5272484ac09ef3e5e36a08f27c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240501 14:05:41.882193 9508 log.cc:1196] T a726769d24284fedadd8e9e63cd27d05 P 1c3fdc5272484ac09ef3e5e36a08f27c: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/wal/wals/a726769d24284fedadd8e9e63cd27d05
I20240501 14:05:41.882891 4838 catalog_manager.cc:4830] TS 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209): tablet a726769d24284fedadd8e9e63cd27d05 (table rebalance_test_table_2 [id=facc024bfd644356a8879fd9661c928d]) successfully deleted
I20240501 14:05:41.883996 5015 tablet_copy_service.cc:342] P 04088ee3bea440a7bd5ff772152b9410: Request end of tablet copy session 1c3fdc5272484ac09ef3e5e36a08f27c-a726769d24284fedadd8e9e63cd27d05 received from {username='slave'} at 127.29.124.131:58573
I20240501 14:05:41.884155 5015 tablet_copy_service.cc:433] P 04088ee3bea440a7bd5ff772152b9410: ending tablet copy session 1c3fdc5272484ac09ef3e5e36a08f27c-a726769d24284fedadd8e9e63cd27d05 on tablet a726769d24284fedadd8e9e63cd27d05 with peer 1c3fdc5272484ac09ef3e5e36a08f27c
I20240501 14:05:41.891321 9659 tablet_bootstrap.cc:492] T 420642af520d45ffbc7f2d35f93e75a7 P 6dac90f01e744f9c8701a404440a4c77: Bootstrap starting.
I20240501 14:05:41.926591 9704 ts_tablet_manager.cc:902] T 8cc5eec4600640f7a293877c825f8d96 P 4360b59e3a154688a79ca5f9c0a37bc9: Initiating tablet copy from peer 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129:44033)
F20240501 14:05:41.927067 9704 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-1a7bbfc15a76450bbc9197037e856e08.queue_time_us, tablet-download-pool-1a7bbfc15a76450bbc9197037e856e08.queue_time_us) (tablet-download-pool-1a7bbfc15a76450bbc9197037e856e08.run_cpu_time_us, tablet-download-pool-1a7bbfc15a76450bbc9197037e856e08.run_cpu_time_us) (tablet-download-pool-1a7bbfc15a76450bbc9197037e856e08.run_wall_time_us, tablet-download-pool-1a7bbfc15a76450bbc9197037e856e08.run_wall_time_us) (tablet-download-pool-51cbcf4888c440c2ad096c988ab2d968.queue_time_us, tablet-download-pool-51cbcf4888c440c2ad096c988ab2d968.queue_time_us) (tablet-download-pool-51cbcf4888c440c2ad096c988ab2d968.run_cpu_time_us, tablet-download-pool-51cbcf4888c440c2ad096c988ab2d968.run_cpu_time_us) (tablet-download-pool-51cbcf4888c440c2ad096c988ab2d968.run_wall_time_us, tablet-download-pool-51cbcf4888c440c2ad096c988ab2d968.run_wall_time_us) (tablet-download-pool-71bdb382b08341218c8bf2459fe65fc0.queue_time_us, tablet-download-pool-71bdb382b08341218c8bf2459fe65fc0.queue_time_us) (tablet-download-pool-71bdb382b08341218c8bf2459fe65fc0.run_cpu_time_us, tablet-download-pool-71bdb382b08341218c8bf2459fe65fc0.run_cpu_time_us) (tablet-download-pool-71bdb382b08341218c8bf2459fe65fc0.run_wall_time_us, tablet-download-pool-71bdb382b08341218c8bf2459fe65fc0.run_wall_time_us) (tablet-download-pool-88ed5438fc934adc9fb98c6cb4476598.queue_time_us, tablet-download-pool-88ed5438fc934adc9fb98c6cb4476598.queue_time_us) (tablet-download-pool-88ed5438fc934adc9fb98c6cb4476598.run_cpu_time_us, tablet-download-pool-88ed5438fc934adc9fb98c6cb4476598.run_cpu_time_us) (tablet-download-pool-88ed5438fc934adc9fb98c6cb4476598.run_wall_time_us, tablet-download-pool-88ed5438fc934adc9fb98c6cb4476598.run_wall_time_us) (tablet-download-pool-8cc5eec4600640f7a293877c825f8d96.queue_time_us, tablet-download-pool-8cc5eec4600640f7a293877c825f8d96.queue_time_us) (tablet-download-pool-99d958617b7e4ef385b94f4c73f4dd84.queue_time_us, tablet-download-pool-99d958617b7e4ef385b94f4c73f4dd84.queue_time_us) (tablet-download-pool-99d958617b7e4ef385b94f4c73f4dd84.run_cpu_time_us, tablet-download-pool-99d958617b7e4ef385b94f4c73f4dd84.run_cpu_time_us) (tablet-download-pool-99d958617b7e4ef385b94f4c73f4dd84.run_wall_time_us, tablet-download-pool-99d958617b7e4ef385b94f4c73f4dd84.run_wall_time_us) (tablet-download-pool-9b88c43cb8704583a3e65c5c35c205f6.queue_time_us, tablet-download-pool-9b88c43cb8704583a3e65c5c35c205f6.queue_time_us) (tablet-download-pool-9b88c43cb8704583a3e65c5c35c205f6.run_cpu_time_us, tablet-download-pool-9b88c43cb8704583a3e65c5c35c205f6.run_cpu_time_us) (tablet-download-pool-9b88c43cb8704583a3e65c5c35c205f6.run_wall_time_us, tablet-download-pool-9b88c43cb8704583a3e65c5c35c205f6.run_wall_time_us) (tablet-download-pool-9fd94635797d447cbdc9181ee91f465b.queue_time_us, tablet-download-pool-9fd94635797d447cbdc9181ee91f465b.queue_time_us) (tablet-download-pool-9fd94635797d447cbdc9181ee91f465b.run_cpu_time_us, tablet-download-pool-9fd94635797d447cbdc9181ee91f465b.run_cpu_time_us) (tablet-download-pool-9fd94635797d447cbdc9181ee91f465b.run_wall_time_us, tablet-download-pool-9fd94635797d447cbdc9181ee91f465b.run_wall_time_us) (tablet-download-pool-ac4b234f794149e4b6def7a3b358942d.queue_time_us, tablet-download-pool-ac4b234f794149e4b6def7a3b358942d.queue_time_us) (tablet-download-pool-ac4b234f794149e4b6def7a3b358942d.run_cpu_time_us, tablet-download-pool-ac4b234f794149e4b6def7a3b358942d.run_cpu_time_us) (tablet-download-pool-ac4b234f794149e4b6def7a3b358942d.run_wall_time_us, tablet-download-pool-ac4b234f794149e4b6def7a3b358942d.run_wall_time_us) (tablet-download-pool-b284aa82594a44f5885f32fa05844913.queue_time_us, tablet-download-pool-b284aa82594a44f5885f32fa05844913.queue_time_us) (tablet-download-pool-b284aa82594a44f5885f32fa05844913.run_cpu_time_us, tablet-download-pool-b284aa82594a44f5885f32fa05844913.run_cpu_time_us) (tablet-download-pool-b284aa82594a44f5885f32fa05844913.run_wall_time_us, tablet-download-pool-b284aa82594a44f5885f32fa05844913.run_wall_time_us) (tablet-download-pool-ba6cba753e934e719c730593c0706346.queue_time_us, tablet-download-pool-ba6cba753e934e719c730593c0706346.queue_time_us) (tablet-download-pool-ba6cba753e934e719c730593c0706346.run_cpu_time_us, tablet-download-pool-ba6cba753e934e719c730593c0706346.run_cpu_time_us) (tablet-download-pool-ba6cba753e934e719c730593c0706346.run_wall_time_us, tablet-download-pool-ba6cba753e934e719c730593c0706346.run_wall_time_us) (tablet-download-pool-c8a961e711114f369ba06840c062b982.queue_time_us, tablet-download-pool-c8a961e711114f369ba06840c062b982.queue_time_us) (tablet-download-pool-c8a961e711114f369ba06840c062b982.run_cpu_time_us, tablet-download-pool-c8a961e711114f369ba06840c062b982.run_cpu_time_us) (tablet-download-pool-c8a961e711114f369ba06840c062b982.run_wall_time_us, tablet-download-pool-c8a961e711114f369ba06840c062b982.run_wall_time_us) (tablet-download-pool-ec1e81011c6a46bc8a39aecf3a5e71e4.queue_time_us, tablet-download-pool-ec1e81011c6a46bc8a39aecf3a5e71e4.queue_time_us) (tablet-download-pool-ec1e81011c6a46bc8a39aecf3a5e71e4.run_cpu_time_us, tablet-download-pool-ec1e81011c6a46bc8a39aecf3a5e71e4.run_cpu_time_us) (tablet-download-pool-ec1e81011c6a46bc8a39aecf3a5e71e4.run_wall_time_us, tablet-download-pool-ec1e81011c6a46bc8a39aecf3a5e71e4.run_wall_time_us) (tablet-download-pool-f4d2b8c005b64bbb8e6621da068d15e5.queue_time_us, tablet-download-pool-f4d2b8c005b64bbb8e6621da068d15e5.queue_time_us) (tablet-download-pool-f4d2b8c005b64bbb8e6621da068d15e5.run_cpu_time_us, tablet-download-pool-f4d2b8c005b64bbb8e6621da068d15e5.run_cpu_time_us) (tablet-download-pool-f4d2b8c005b64bbb8e6621da068d15e5.run_wall_time_us, tablet-download-pool-f4d2b8c005b64bbb8e6621da068d15e5.run_wall_time_us) (tablet-download-pool-fee16905387d428aa02cd2070c86c790.queue_time_us, tablet-download-pool-fee16905387d428aa02cd2070c86c790.queue_time_us) (tablet-download-pool-fee16905387d428aa02cd2070c86c790.run_cpu_time_us, tablet-download-pool-fee16905387d428aa02cd2070c86c790.run_cpu_time_us) (tablet-download-pool-fee16905387d428aa02cd2070c86c790.run_wall_time_us, tablet-download-pool-fee16905387d428aa02cd2070c86c790.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
I20240501 14:05:41.951613 9701 tablet_replica.cc:330] T f9cb784f60774ea7b6e70ccfebf867a8 P be56773105e744919601127a94e98adf: stopping tablet replica
I20240501 14:05:41.951965 9701 raft_consensus.cc:2229] T f9cb784f60774ea7b6e70ccfebf867a8 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:05:41.952333 9701 raft_consensus.cc:2258] T f9cb784f60774ea7b6e70ccfebf867a8 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:05:41.980541 9659 tablet_bootstrap.cc:492] T 420642af520d45ffbc7f2d35f93e75a7 P 6dac90f01e744f9c8701a404440a4c77: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:05:41.980971 9659 tablet_bootstrap.cc:492] T 420642af520d45ffbc7f2d35f93e75a7 P 6dac90f01e744f9c8701a404440a4c77: Bootstrap complete.
I20240501 14:05:41.981276 9659 ts_tablet_manager.cc:1367] T 420642af520d45ffbc7f2d35f93e75a7 P 6dac90f01e744f9c8701a404440a4c77: Time spent bootstrapping tablet: real 0.091s user 0.050s sys 0.000s
I20240501 14:05:41.981984 9659 raft_consensus.cc:348] T 420642af520d45ffbc7f2d35f93e75a7 P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } }
I20240501 14:05:41.982236 9659 raft_consensus.cc:729] T 420642af520d45ffbc7f2d35f93e75a7 P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6dac90f01e744f9c8701a404440a4c77, State: Initialized, Role: LEARNER
I20240501 14:05:41.982455 9701 ts_tablet_manager.cc:1875] T f9cb784f60774ea7b6e70ccfebf867a8 P be56773105e744919601127a94e98adf: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:05:41.982451 9659 consensus_queue.cc:260] T 420642af520d45ffbc7f2d35f93e75a7 P 6dac90f01e744f9c8701a404440a4c77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } }
I20240501 14:05:41.983032 9659 ts_tablet_manager.cc:1398] T 420642af520d45ffbc7f2d35f93e75a7 P 6dac90f01e744f9c8701a404440a4c77: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:05:41.994187 5406 tablet_copy_service.cc:342] P be56773105e744919601127a94e98adf: Request end of tablet copy session 6dac90f01e744f9c8701a404440a4c77-420642af520d45ffbc7f2d35f93e75a7 received from {username='slave'} at 127.29.124.136:50719
I20240501 14:05:41.994344 5406 tablet_copy_service.cc:433] P be56773105e744919601127a94e98adf: ending tablet copy session 6dac90f01e744f9c8701a404440a4c77-420642af520d45ffbc7f2d35f93e75a7 on tablet 420642af520d45ffbc7f2d35f93e75a7 with peer 6dac90f01e744f9c8701a404440a4c77
@ 0x7ff5615ebdcd google::LogMessage::Fail() at ??:0
@ 0x7ff5615efb93 google::LogMessage::SendToLog() at ??:0
@ 0x7ff5615eb7cc google::LogMessage::Flush() at ??:0
I20240501 14:05:42.026983 9701 ts_tablet_manager.cc:1888] T f9cb784f60774ea7b6e70ccfebf867a8 P be56773105e744919601127a94e98adf: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 14:05:42.027146 9701 log.cc:1196] T f9cb784f60774ea7b6e70ccfebf867a8 P be56773105e744919601127a94e98adf: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-3/wal/wals/f9cb784f60774ea7b6e70ccfebf867a8
I20240501 14:05:42.028522 4839 catalog_manager.cc:4830] TS be56773105e744919601127a94e98adf (127.29.124.132:35459): tablet f9cb784f60774ea7b6e70ccfebf867a8 (table rebalance_test_table_5 [id=3794b1ccb5004f7cb92a229ef6ce3595]) successfully deleted
I20240501 14:05:42.032630 6888 raft_consensus.cc:2932] T 8cc5eec4600640f7a293877c825f8d96 P 938a24b0e6a040b7966225af26558ff8 [term 1 LEARNER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:42.035465 4973 tablet_service.cc:1429] Tablet server has 6 leaders and 0 scanners
I20240501 14:05:42.048231 7285 raft_consensus.cc:1203] T 420642af520d45ffbc7f2d35f93e75a7 P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
@ 0x7ff5615ecf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240501 14:05:42.087410 7267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
@ 0x7ff56432c742 kudu::TraceMetrics::InternName() at ??:0
I20240501 14:05:42.097739 9690 raft_consensus.cc:1050] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4: attempting to promote NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 to VOTER
I20240501 14:05:42.098620 9690 consensus_queue.cc:237] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } }
I20240501 14:05:42.108485 7136 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
@ 0x7ff5642f8f8b kudu::ThreadPool::ThreadPool() at ??:0
I20240501 14:05:42.111984 7004 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:42.131599 6888 raft_consensus.cc:1261] T 3d6859d53fd747a5a5e94722626fa522 P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:05:42.132272 9690 consensus_queue.cc:1035] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:05:42.132516 7025 raft_consensus.cc:1261] T 3d6859d53fd747a5a5e94722626fa522 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 LEARNER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:05:42.133076 9688 consensus_queue.cc:1035] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
@ 0x7ff5642f2cd6 kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7ff5737458de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240501 14:05:42.141142 6872 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
@ 0x7ff57375ddaf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240501 14:05:42.161604 5365 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:42.169023 5254 raft_consensus.cc:1261] T 3d6859d53fd747a5a5e94722626fa522 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:05:42.169585 9690 consensus_queue.cc:1035] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
@ 0x7ff573888480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240501 14:05:42.193112 5236 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:42.193742 5106 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:42.200317 9333 tablet_bootstrap.cc:492] T 2c39b53867e047faaf24cc6f87a8100c P 19a6d2d8a08142d5bffcde3a37234692: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:05:42.200745 9333 tablet_bootstrap.cc:492] T 2c39b53867e047faaf24cc6f87a8100c P 19a6d2d8a08142d5bffcde3a37234692: Bootstrap complete.
I20240501 14:05:42.201020 9333 ts_tablet_manager.cc:1367] T 2c39b53867e047faaf24cc6f87a8100c P 19a6d2d8a08142d5bffcde3a37234692: Time spent bootstrapping tablet: real 0.583s user 0.065s sys 0.041s
@ 0x7ff5738c0a8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240501 14:05:42.221221 9333 raft_consensus.cc:348] T 2c39b53867e047faaf24cc6f87a8100c P 19a6d2d8a08142d5bffcde3a37234692 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } }
I20240501 14:05:42.221549 9333 raft_consensus.cc:729] T 2c39b53867e047faaf24cc6f87a8100c P 19a6d2d8a08142d5bffcde3a37234692 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 19a6d2d8a08142d5bffcde3a37234692, State: Initialized, Role: LEARNER
I20240501 14:05:42.221762 9333 consensus_queue.cc:260] T 2c39b53867e047faaf24cc6f87a8100c P 19a6d2d8a08142d5bffcde3a37234692 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } }
I20240501 14:05:42.222368 9333 ts_tablet_manager.cc:1398] T 2c39b53867e047faaf24cc6f87a8100c P 19a6d2d8a08142d5bffcde3a37234692: Time spent starting tablet: real 0.021s user 0.001s sys 0.003s
I20240501 14:05:42.228221 5406 tablet_copy_service.cc:342] P be56773105e744919601127a94e98adf: Request end of tablet copy session 19a6d2d8a08142d5bffcde3a37234692-2c39b53867e047faaf24cc6f87a8100c received from {username='slave'} at 127.29.124.134:56125
I20240501 14:05:42.228359 5406 tablet_copy_service.cc:433] P be56773105e744919601127a94e98adf: ending tablet copy session 19a6d2d8a08142d5bffcde3a37234692-2c39b53867e047faaf24cc6f87a8100c on tablet 2c39b53867e047faaf24cc6f87a8100c with peer 19a6d2d8a08142d5bffcde3a37234692
@ 0x7ff5738a3981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7ff5738a370e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
I20240501 14:05:42.243014 9690 raft_consensus.cc:2932] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
@ 0x7ff5642fcb75 kudu::ThreadPool::DispatchThread() at ??:0
I20240501 14:05:42.245946 9742 ts_tablet_manager.cc:902] T e7d5f02ccba34254a4d20951878b1214 P 938a24b0e6a040b7966225af26558ff8: Initiating tablet copy from peer be56773105e744919601127a94e98adf (127.29.124.132:35459)
I20240501 14:05:42.246337 5255 consensus_queue.cc:237] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }
I20240501 14:05:42.251969 4847 catalog_manager.cc:5484] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 reported cstate change: config changed from index 7 to 8, 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:05:42.258273 7025 raft_consensus.cc:2932] T 3d6859d53fd747a5a5e94722626fa522 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
@ 0x7ff56430019c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
I20240501 14:05:42.270812 7023 raft_consensus.cc:1261] T 404e2851055541f0be1bb0971a7cc72a P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 14:05:42.271320 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 404e2851055541f0be1bb0971a7cc72a. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:42.271883 9576 consensus_queue.cc:1035] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:05:42.279935 5254 raft_consensus.cc:2932] T 3d6859d53fd747a5a5e94722626fa522 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:42.265373 5125 consensus_queue.cc:237] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } }
I20240501 14:05:42.288750 7023 raft_consensus.cc:1261] T 3d6859d53fd747a5a5e94722626fa522 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:05:42.291594 6888 raft_consensus.cc:1261] T 3d6859d53fd747a5a5e94722626fa522 P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
@ 0x7ff564300019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
I20240501 14:05:42.304515 9520 consensus_queue.cc:1035] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.014s
I20240501 14:05:42.307210 5125 raft_consensus.cc:1261] T 404e2851055541f0be1bb0971a7cc72a P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:05:42.309542 6888 raft_consensus.cc:2932] T 3d6859d53fd747a5a5e94722626fa522 P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:42.310457 9750 consensus_queue.cc:1035] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:05:42.315440 9688 consensus_queue.cc:1035] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:05:42.318389 9520 raft_consensus.cc:2932] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:42.322899 9576 raft_consensus.cc:2932] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:42.324388 4839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 3d6859d53fd747a5a5e94722626fa522 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:05:42.326373 4847 catalog_manager.cc:5484] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 reported cstate change: config changed from index 8 to 9, VOTER 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) evicted. New cstate: current_term: 1 leader_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:05:42.326313 7023 raft_consensus.cc:2932] T 404e2851055541f0be1bb0971a7cc72a P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:42.330981 5236 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d6859d53fd747a5a5e94722626fa522 with delete_type TABLET_DATA_TOMBSTONED (TS 1c3fdc5272484ac09ef3e5e36a08f27c not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:39020
I20240501 14:05:42.331379 9508 tablet_replica.cc:330] T 3d6859d53fd747a5a5e94722626fa522 P 1c3fdc5272484ac09ef3e5e36a08f27c: stopping tablet replica
I20240501 14:05:42.331621 9508 raft_consensus.cc:2229] T 3d6859d53fd747a5a5e94722626fa522 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:05:42.331900 9508 raft_consensus.cc:2258] T 3d6859d53fd747a5a5e94722626fa522 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Raft consensus is shut down!
I20240501 14:05:42.332123 4854 catalog_manager.cc:5484] T 404e2851055541f0be1bb0971a7cc72a P 19a6d2d8a08142d5bffcde3a37234692 reported cstate change: config changed from index 6 to 7, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New cstate: current_term: 1 leader_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
I20240501 14:05:42.334017 9508 ts_tablet_manager.cc:1875] T 3d6859d53fd747a5a5e94722626fa522 P 1c3fdc5272484ac09ef3e5e36a08f27c: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:05:42.339063 7023 raft_consensus.cc:1203] T 2c39b53867e047faaf24cc6f87a8100c P 19a6d2d8a08142d5bffcde3a37234692 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.11->[1.12-1.12] Dedup: 1.12->[]
I20240501 14:05:42.356453 9508 ts_tablet_manager.cc:1888] T 3d6859d53fd747a5a5e94722626fa522 P 1c3fdc5272484ac09ef3e5e36a08f27c: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240501 14:05:42.356595 9508 log.cc:1196] T 3d6859d53fd747a5a5e94722626fa522 P 1c3fdc5272484ac09ef3e5e36a08f27c: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-2/wal/wals/3d6859d53fd747a5a5e94722626fa522
I20240501 14:05:42.365438 6888 raft_consensus.cc:2932] T 3d6859d53fd747a5a5e94722626fa522 P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:42.366921 7025 raft_consensus.cc:2932] T 3d6859d53fd747a5a5e94722626fa522 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:42.378048 9756 ts_tablet_manager.cc:902] T 9d32e0707d244f2c9bbcb57f62b09f31 P 4360b59e3a154688a79ca5f9c0a37bc9: Initiating tablet copy from peer e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377)
I20240501 14:05:42.382129 4838 catalog_manager.cc:4830] TS 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209): tablet 3d6859d53fd747a5a5e94722626fa522 (table rebalance_test_table_8 [id=f26ba2f4e5654a5fa72021df5929200d]) successfully deleted
I20240501 14:05:42.383744 5125 raft_consensus.cc:2932] T 404e2851055541f0be1bb0971a7cc72a P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } } }
@ 0x7ff5642db01e kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff5676606db start_thread at ??:0
I20240501 14:05:42.408172 9742 tablet_copy_client.cc:308] T e7d5f02ccba34254a4d20951878b1214 P 938a24b0e6a040b7966225af26558ff8: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.132:35459
I20240501 14:05:42.408918 5406 tablet_copy_service.cc:140] P be56773105e744919601127a94e98adf: Received BeginTabletCopySession request for tablet e7d5f02ccba34254a4d20951878b1214 from peer 938a24b0e6a040b7966225af26558ff8 ({username='slave'} at 127.29.124.133:50583)
I20240501 14:05:42.409096 5406 tablet_copy_service.cc:161] P be56773105e744919601127a94e98adf: Beginning new tablet copy session on tablet e7d5f02ccba34254a4d20951878b1214 from peer 938a24b0e6a040b7966225af26558ff8 at {username='slave'} at 127.29.124.133:50583: session id = 938a24b0e6a040b7966225af26558ff8-e7d5f02ccba34254a4d20951878b1214
I20240501 14:05:42.412201 5406 tablet_copy_source_session.cc:205] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:05:42.413332 9742 ts_tablet_manager.cc:1875] T e7d5f02ccba34254a4d20951878b1214 P 938a24b0e6a040b7966225af26558ff8: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 14:05:42.415246 5254 consensus_queue.cc:237] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false replace: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } }
W20240501 14:05:42.418170 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1a7bbfc15a76450bbc9197037e856e08. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:42.418664 7025 raft_consensus.cc:1261] T 1a7bbfc15a76450bbc9197037e856e08 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:05:42.419464 9767 consensus_queue.cc:1035] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:05:42.420795 7155 raft_consensus.cc:1261] T 1a7bbfc15a76450bbc9197037e856e08 P 4360b59e3a154688a79ca5f9c0a37bc9 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:05:42.421871 9764 consensus_queue.cc:1035] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:05:42.425472 4993 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "a726769d24284fedadd8e9e63cd27d05"
dest_uuid: "04088ee3bea440a7bd5ff772152b9410"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51272
I20240501 14:05:42.425740 4993 raft_consensus.cc:595] T a726769d24284fedadd8e9e63cd27d05 P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:05:42.430267 9742 ts_tablet_manager.cc:1888] T e7d5f02ccba34254a4d20951878b1214 P 938a24b0e6a040b7966225af26558ff8: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.10
I20240501 14:05:42.430687 9742 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7d5f02ccba34254a4d20951878b1214. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7ff56049e71f clone at ??:0
I20240501 14:05:42.455760 9742 tablet_copy_client.cc:783] T e7d5f02ccba34254a4d20951878b1214 P 938a24b0e6a040b7966225af26558ff8: tablet copy: Starting download of 0 data blocks...
I20240501 14:05:42.456018 9742 tablet_copy_client.cc:647] T e7d5f02ccba34254a4d20951878b1214 P 938a24b0e6a040b7966225af26558ff8: tablet copy: Starting download of 1 WAL segments...
I20240501 14:05:42.461830 8375 raft_consensus.cc:1050] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410: attempting to promote NON_VOTER 938a24b0e6a040b7966225af26558ff8 to VOTER
I20240501 14:05:42.462697 8375 consensus_queue.cc:237] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } }
I20240501 14:05:42.466028 6888 raft_consensus.cc:1261] T 8cc5eec4600640f7a293877c825f8d96 P 938a24b0e6a040b7966225af26558ff8 [term 1 LEARNER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:05:42.466615 9658 consensus_queue.cc:1035] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:05:42.473583 9742 tablet_copy_client.cc:523] T e7d5f02ccba34254a4d20951878b1214 P 938a24b0e6a040b7966225af26558ff8: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:05:42.475495 7023 raft_consensus.cc:1261] T 8cc5eec4600640f7a293877c825f8d96 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:05:42.475540 5125 raft_consensus.cc:1261] T 8cc5eec4600640f7a293877c825f8d96 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:05:42.476156 9658 consensus_queue.cc:1035] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:05:42.476413 8375 consensus_queue.cc:1035] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:05:42.507400 9764 raft_consensus.cc:2932] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false replace: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:42.508019 7024 raft_consensus.cc:2932] T 1a7bbfc15a76450bbc9197037e856e08 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false replace: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:42.517035 4854 catalog_manager.cc:5484] T 1a7bbfc15a76450bbc9197037e856e08 P 19a6d2d8a08142d5bffcde3a37234692 reported cstate change: config changed from index 9 to 10, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New cstate: current_term: 1 leader_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false replace: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:05:42.531224 9742 tablet_bootstrap.cc:492] T e7d5f02ccba34254a4d20951878b1214 P 938a24b0e6a040b7966225af26558ff8: Bootstrap starting.
I20240501 14:05:42.534891 8375 raft_consensus.cc:2932] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:42.539945 6888 raft_consensus.cc:2932] T 8cc5eec4600640f7a293877c825f8d96 P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:42.546799 4854 catalog_manager.cc:5484] T 8cc5eec4600640f7a293877c825f8d96 P 938a24b0e6a040b7966225af26558ff8 reported cstate change: config changed from index 10 to 11, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "04088ee3bea440a7bd5ff772152b9410" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:42.549341 7023 raft_consensus.cc:2932] T 8cc5eec4600640f7a293877c825f8d96 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:42.552012 5126 raft_consensus.cc:2932] T 8cc5eec4600640f7a293877c825f8d96 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: NON_VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: true } } }
I20240501 14:05:42.563890 4993 consensus_queue.cc:237] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:05:42.566771 7024 raft_consensus.cc:1261] T 8cc5eec4600640f7a293877c825f8d96 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:05:42.567342 9658 consensus_queue.cc:1035] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:05:42.570209 6888 raft_consensus.cc:1261] T 8cc5eec4600640f7a293877c825f8d96 P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:05:42.570782 9658 consensus_queue.cc:1035] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:05:42.578303 7024 raft_consensus.cc:2932] T 8cc5eec4600640f7a293877c825f8d96 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:42.584192 4854 catalog_manager.cc:5484] T 8cc5eec4600640f7a293877c825f8d96 P 19a6d2d8a08142d5bffcde3a37234692 reported cstate change: config changed from index 11 to 12, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) evicted. New cstate: current_term: 1 leader_uuid: "04088ee3bea440a7bd5ff772152b9410" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:42.586176 9658 raft_consensus.cc:2932] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:42.595649 4839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8cc5eec4600640f7a293877c825f8d96 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:05:42.609248 9616 raft_consensus.cc:1050] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf: attempting to promote NON_VOTER 6dac90f01e744f9c8701a404440a4c77 to VOTER
I20240501 14:05:42.610139 9616 consensus_queue.cc:237] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } }
I20240501 14:05:42.613440 5255 raft_consensus.cc:1261] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:05:42.614087 8891 consensus_queue.cc:1035] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:05:42.615392 5126 raft_consensus.cc:1261] T 8cc5eec4600640f7a293877c825f8d96 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:05:42.616379 7285 raft_consensus.cc:1261] T 420642af520d45ffbc7f2d35f93e75a7 P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:05:42.617502 8891 consensus_queue.cc:1035] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:05:42.621047 6889 raft_consensus.cc:2932] T 8cc5eec4600640f7a293877c825f8d96 P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:42.628726 9784 consensus_queue.cc:1035] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.001s
I20240501 14:05:42.634269 6888 raft_consensus.cc:1261] T 420642af520d45ffbc7f2d35f93e75a7 P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:05:42.634878 8891 consensus_queue.cc:1035] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:05:42.635109 8375 raft_consensus.cc:979] T a726769d24284fedadd8e9e63cd27d05 P 04088ee3bea440a7bd5ff772152b9410: : Instructing follower 19a6d2d8a08142d5bffcde3a37234692 to start an election
I20240501 14:05:42.635805 8375 raft_consensus.cc:1067] T a726769d24284fedadd8e9e63cd27d05 P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEADER]: Signalling peer 19a6d2d8a08142d5bffcde3a37234692 to start an election
I20240501 14:05:42.637151 7024 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "a726769d24284fedadd8e9e63cd27d05"
dest_uuid: "19a6d2d8a08142d5bffcde3a37234692"
from {username='slave'} at 127.29.124.129:57543
I20240501 14:05:42.637775 7024 raft_consensus.cc:482] T a726769d24284fedadd8e9e63cd27d05 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:05:42.637969 7024 raft_consensus.cc:3037] T a726769d24284fedadd8e9e63cd27d05 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:05:42.640724 7024 raft_consensus.cc:504] T a726769d24284fedadd8e9e63cd27d05 P 19a6d2d8a08142d5bffcde3a37234692 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } }
I20240501 14:05:42.642164 8891 raft_consensus.cc:2932] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:42.643096 5126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a726769d24284fedadd8e9e63cd27d05" candidate_uuid: "19a6d2d8a08142d5bffcde3a37234692" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "e9c60449cf3c46f8aca4ab6c535302a4"
I20240501 14:05:42.643299 5126 raft_consensus.cc:3037] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:05:42.645882 5126 raft_consensus.cc:2454] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19a6d2d8a08142d5bffcde3a37234692 in term 2.
I20240501 14:05:42.645956 5255 raft_consensus.cc:2932] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:42.648780 4994 consensus_queue.cc:237] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:05:42.653609 4847 catalog_manager.cc:5484] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c reported cstate change: config changed from index 7 to 8, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:42.659927 6889 raft_consensus.cc:1261] T 8cc5eec4600640f7a293877c825f8d96 P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:05:42.661350 9658 consensus_queue.cc:1035] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:05:42.661502 7025 raft_consensus.cc:1261] T 8cc5eec4600640f7a293877c825f8d96 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:05:42.663650 9658 consensus_queue.cc:1035] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:05:42.664894 6889 raft_consensus.cc:2932] T 8cc5eec4600640f7a293877c825f8d96 P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:42.668749 9784 raft_consensus.cc:2932] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:42.671605 4847 catalog_manager.cc:5484] T 8cc5eec4600640f7a293877c825f8d96 P 938a24b0e6a040b7966225af26558ff8 reported cstate change: config changed from index 12 to 13, VOTER e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130) evicted. New cstate: current_term: 1 leader_uuid: "04088ee3bea440a7bd5ff772152b9410" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:42.675529 7025 raft_consensus.cc:2932] T 8cc5eec4600640f7a293877c825f8d96 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, VOTER e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130) evicted. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } }
I20240501 14:05:42.676555 4994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a726769d24284fedadd8e9e63cd27d05" candidate_uuid: "19a6d2d8a08142d5bffcde3a37234692" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "04088ee3bea440a7bd5ff772152b9410"
I20240501 14:05:42.683115 4839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8cc5eec4600640f7a293877c825f8d96 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240501 14:05:42.798565 5191 connection.cc:535] client connection to 127.29.124.135:35329 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:05:42.930738 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:05:42.930691 4839 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.ChangeConfig -> {remote=127.29.124.129:44033, user_credentials={real_user=slave}} blocked reactor thread for 247559us
W20240501 14:05:42.799283 4840 connection.cc:535] client connection to 127.29.124.135:35329 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:05:42.931270 4840 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:05:42.891268 4827 kernel_stack_watchdog.cc:198] Thread 4839 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:308 for 164ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fdf158b2980 (unknown) at ??:0
@ 0x7fdf0e6d4255 __write at ??:0
@ 0x7fdf0e64f28d _IO_file_write at ??:0
@ 0x7fdf0e64fbff _IO_file_xsputn at ??:0
@ 0x7fdf0e643a57 fwrite at ??:0
@ 0x8793e7 __interceptor_fwrite.part.56 at ??:?
@ 0x7fdf0f8368a0 google::LogMessage::SendToLog() at ??:0
@ 0x7fdf0f8327cc google::LogMessage::Flush() at ??:0
@ 0x7fdf0f832a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7fdf2592dbb1 kudu::master::AsyncChangeConfigTask::HandleResponse() at ??:0
@ 0x7fdf2592ab09 kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7fdf259694b0 kudu::master::AsyncEvictReplicaTask::SendRequest()::$_33::operator()() at ??:0
@ 0x7fdf259692d9 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7fdf191728cf kudu::rpc::Proxy::AsyncRequest()::$_1::operator()() at ??:0
@ 0x7fdf191724ae std::_Function_handler<>::_M_invoke() at ??:0
W20240501 14:05:42.957370 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:05:42.963853 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:05:42.964113 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:05:42.965759 5191 consensus_peers.cc:485] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:05:42.968717 4840 catalog_manager.cc:4573] TS 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 8cc5eec4600640f7a293877c825f8d96: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
I20240501 14:05:42.694978 7285 raft_consensus.cc:2932] T 420642af520d45ffbc7f2d35f93e75a7 P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:42.682758 5106 tablet_service.cc:1514] Processing DeleteTablet for tablet 8cc5eec4600640f7a293877c825f8d96 with delete_type TABLET_DATA_TOMBSTONED (TS e9c60449cf3c46f8aca4ab6c535302a4 not found in new config with opid_index 13) from {username='slave'} at 127.0.0.1:41850
I20240501 14:05:42.676749 4994 raft_consensus.cc:3032] T a726769d24284fedadd8e9e63cd27d05 P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:05:42.727465 5126 raft_consensus.cc:1226] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 [term 2 FOLLOWER]: Rejecting Update request from peer 04088ee3bea440a7bd5ff772152b9410 for earlier term 1. Current term is 2. Ops: []
I20240501 14:05:42.993247 4994 raft_consensus.cc:729] T a726769d24284fedadd8e9e63cd27d05 P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 04088ee3bea440a7bd5ff772152b9410, State: Running, Role: LEADER
W20240501 14:05:42.800020 5061 connection.cc:535] client connection to 127.29.124.135:35329 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:05:42.993947 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:05:42.993950 4994 consensus_queue.cc:260] T a726769d24284fedadd8e9e63cd27d05 P 04088ee3bea440a7bd5ff772152b9410 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } }
W20240501 14:05:42.800246 4930 connection.cc:535] client connection to 127.29.124.135:35329 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:05:42.994402 4994 raft_consensus.cc:3037] T a726769d24284fedadd8e9e63cd27d05 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:05:42.994495 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:05:42.958514 9899 ts_tablet_manager.cc:902] T 1a7bbfc15a76450bbc9197037e856e08 P 6dac90f01e744f9c8701a404440a4c77: Initiating tablet copy from peer 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209)
I20240501 14:05:42.997123 4994 raft_consensus.cc:2454] T a726769d24284fedadd8e9e63cd27d05 P 04088ee3bea440a7bd5ff772152b9410 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19a6d2d8a08142d5bffcde3a37234692 in term 2.
W20240501 14:05:42.998072 9817 consensus_queue.cc:1159] T a726769d24284fedadd8e9e63cd27d05 P 04088ee3bea440a7bd5ff772152b9410 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" responder_term: 2 status { last_received { term: 1 index: 7 } last_committed_idx: 7 error { code: INVALID_TERM status { code: ILLEGAL_STATE message: "Rejecting Update request from peer 04088ee3bea440a7bd5ff772152b9410 for earlier term 1. Current term is 2. Ops: []" } } last_received_current_leader { term: 0 index: 0 } }
I20240501 14:05:43.016986 9910 tablet_replica.cc:330] T 8cc5eec4600640f7a293877c825f8d96 P e9c60449cf3c46f8aca4ab6c535302a4: stopping tablet replica
I20240501 14:05:43.020516 9910 raft_consensus.cc:2229] T 8cc5eec4600640f7a293877c825f8d96 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240501 14:05:43.029599 9899 tablet_copy_client.cc:308] T 1a7bbfc15a76450bbc9197037e856e08 P 6dac90f01e744f9c8701a404440a4c77: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.131:33209
W20240501 14:05:43.021283 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:05:43.050060 4930 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:05:43.050237 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:43.051172 5276 tablet_copy_service.cc:140] P 1c3fdc5272484ac09ef3e5e36a08f27c: Received BeginTabletCopySession request for tablet 1a7bbfc15a76450bbc9197037e856e08 from peer 6dac90f01e744f9c8701a404440a4c77 ({username='slave'} at 127.29.124.136:49141)
I20240501 14:05:43.051368 5276 tablet_copy_service.cc:161] P 1c3fdc5272484ac09ef3e5e36a08f27c: Beginning new tablet copy session on tablet 1a7bbfc15a76450bbc9197037e856e08 from peer 6dac90f01e744f9c8701a404440a4c77 at {username='slave'} at 127.29.124.136:49141: session id = 6dac90f01e744f9c8701a404440a4c77-1a7bbfc15a76450bbc9197037e856e08
I20240501 14:05:43.054174 5276 tablet_copy_source_session.cc:205] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c: Tablet Copy: opened 0 blocks and 1 log segments
I20240501 14:05:43.057466 9899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a7bbfc15a76450bbc9197037e856e08. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:05:42.719890 6888 raft_consensus.cc:2932] T 420642af520d45ffbc7f2d35f93e75a7 P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:43.066977 9899 tablet_copy_client.cc:783] T 1a7bbfc15a76450bbc9197037e856e08 P 6dac90f01e744f9c8701a404440a4c77: tablet copy: Starting download of 0 data blocks...
I20240501 14:05:43.067247 9899 tablet_copy_client.cc:647] T 1a7bbfc15a76450bbc9197037e856e08 P 6dac90f01e744f9c8701a404440a4c77: tablet copy: Starting download of 1 WAL segments...
I20240501 14:05:42.803371 9842 ts_tablet_manager.cc:902] T 404e2851055541f0be1bb0971a7cc72a P 938a24b0e6a040b7966225af26558ff8: Initiating tablet copy from peer 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209)
I20240501 14:05:42.851351 9742 tablet_bootstrap.cc:492] T e7d5f02ccba34254a4d20951878b1214 P 938a24b0e6a040b7966225af26558ff8: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:05:43.068678 9742 tablet_bootstrap.cc:492] T e7d5f02ccba34254a4d20951878b1214 P 938a24b0e6a040b7966225af26558ff8: Bootstrap complete.
F20240501 14:05:43.068604 9842 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-2c39b53867e047faaf24cc6f87a8100c.queue_time_us, tablet-download-pool-2c39b53867e047faaf24cc6f87a8100c.queue_time_us) (tablet-download-pool-2c39b53867e047faaf24cc6f87a8100c.run_cpu_time_us, tablet-download-pool-2c39b53867e047faaf24cc6f87a8100c.run_cpu_time_us) (tablet-download-pool-2c39b53867e047faaf24cc6f87a8100c.run_wall_time_us, tablet-download-pool-2c39b53867e047faaf24cc6f87a8100c.run_wall_time_us) (tablet-download-pool-3d6859d53fd747a5a5e94722626fa522.queue_time_us, tablet-download-pool-3d6859d53fd747a5a5e94722626fa522.queue_time_us) (tablet-download-pool-3d6859d53fd747a5a5e94722626fa522.run_cpu_time_us, tablet-download-pool-3d6859d53fd747a5a5e94722626fa522.run_cpu_time_us) (tablet-download-pool-3d6859d53fd747a5a5e94722626fa522.run_wall_time_us, tablet-download-pool-3d6859d53fd747a5a5e94722626fa522.run_wall_time_us) (tablet-download-pool-404e2851055541f0be1bb0971a7cc72a.queue_time_us, tablet-download-pool-404e2851055541f0be1bb0971a7cc72a.queue_time_us) (tablet-download-pool-420642af520d45ffbc7f2d35f93e75a7.queue_time_us, tablet-download-pool-420642af520d45ffbc7f2d35f93e75a7.queue_time_us) (tablet-download-pool-420642af520d45ffbc7f2d35f93e75a7.run_cpu_time_us, tablet-download-pool-420642af520d45ffbc7f2d35f93e75a7.run_cpu_time_us) (tablet-download-pool-420642af520d45ffbc7f2d35f93e75a7.run_wall_time_us, tablet-download-pool-420642af520d45ffbc7f2d35f93e75a7.run_wall_time_us) (tablet-download-pool-51cbcf4888c440c2ad096c988ab2d968.queue_time_us, tablet-download-pool-51cbcf4888c440c2ad096c988ab2d968.queue_time_us) (tablet-download-pool-51cbcf4888c440c2ad096c988ab2d968.run_cpu_time_us, tablet-download-pool-51cbcf4888c440c2ad096c988ab2d968.run_cpu_time_us) (tablet-download-pool-51cbcf4888c440c2ad096c988ab2d968.run_wall_time_us, tablet-download-pool-51cbcf4888c440c2ad096c988ab2d968.run_wall_time_us) (tablet-download-pool-61a274b24f53458f88c2cde4f8dbe4f4.queue_time_us, tablet-download-pool-61a274b24f53458f88c2cde4f8dbe4f4.queue_time_us) (tablet-download-pool-61a274b24f53458f88c2cde4f8dbe4f4.run_cpu_time_us, tablet-download-pool-61a274b24f53458f88c2cde4f8dbe4f4.run_cpu_time_us) (tablet-download-pool-61a274b24f53458f88c2cde4f8dbe4f4.run_wall_time_us, tablet-download-pool-61a274b24f53458f88c2cde4f8dbe4f4.run_wall_time_us) (tablet-download-pool-63650547ab444f45bb377bcb49f258ad.queue_time_us, tablet-download-pool-63650547ab444f45bb377bcb49f258ad.queue_time_us) (tablet-download-pool-63650547ab444f45bb377bcb49f258ad.run_cpu_time_us, tablet-download-pool-63650547ab444f45bb377bcb49f258ad.run_cpu_time_us) (tablet-download-pool-63650547ab444f45bb377bcb49f258ad.run_wall_time_us, tablet-download-pool-63650547ab444f45bb377bcb49f258ad.run_wall_time_us) (tablet-download-pool-71660575d94748d8a9329f909a42a0cd.queue_time_us, tablet-download-pool-71660575d94748d8a9329f909a42a0cd.queue_time_us) (tablet-download-pool-71660575d94748d8a9329f909a42a0cd.run_cpu_time_us, tablet-download-pool-71660575d94748d8a9329f909a42a0cd.run_cpu_time_us) (tablet-download-pool-71660575d94748d8a9329f909a42a0cd.run_wall_time_us, tablet-download-pool-71660575d94748d8a9329f909a42a0cd.run_wall_time_us) (tablet-download-pool-8cc5eec4600640f7a293877c825f8d96.queue_time_us, tablet-download-pool-8cc5eec4600640f7a293877c825f8d96.queue_time_us) (tablet-download-pool-8cc5eec4600640f7a293877c825f8d96.run_cpu_time_us, tablet-download-pool-8cc5eec4600640f7a293877c825f8d96.run_cpu_time_us) (tablet-download-pool-8cc5eec4600640f7a293877c825f8d96.run_wall_time_us, tablet-download-pool-8cc5eec4600640f7a293877c825f8d96.run_wall_time_us) (tablet-download-pool-9b88c43cb8704583a3e65c5c35c205f6.queue_time_us, tablet-download-pool-9b88c43cb8704583a3e65c5c35c205f6.queue_time_us) (tablet-download-pool-9b88c43cb8704583a3e65c5c35c205f6.run_cpu_time_us, tablet-download-pool-9b88c43cb8704583a3e65c5c35c205f6.run_cpu_time_us) (tablet-download-pool-9b88c43cb8704583a3e65c5c35c205f6.run_wall_time_us, tablet-download-pool-9b88c43cb8704583a3e65c5c35c205f6.run_wall_time_us) (tablet-download-pool-b284aa82594a44f5885f32fa05844913.queue_time_us, tablet-download-pool-b284aa82594a44f5885f32fa05844913.queue_time_us) (tablet-download-pool-b284aa82594a44f5885f32fa05844913.run_cpu_time_us, tablet-download-pool-b284aa82594a44f5885f32fa05844913.run_cpu_time_us) (tablet-download-pool-b284aa82594a44f5885f32fa05844913.run_wall_time_us, tablet-download-pool-b284aa82594a44f5885f32fa05844913.run_wall_time_us) (tablet-download-pool-c8a961e711114f369ba06840c062b982.queue_time_us, tablet-download-pool-c8a961e711114f369ba06840c062b982.queue_time_us) (tablet-download-pool-c8a961e711114f369ba06840c062b982.run_cpu_time_us, tablet-download-pool-c8a961e711114f369ba06840c062b982.run_cpu_time_us) (tablet-download-pool-c8a961e711114f369ba06840c062b982.run_wall_time_us, tablet-download-pool-c8a961e711114f369ba06840c062b982.run_wall_time_us) (tablet-download-pool-de6c7753c8564107b858876fcfa5924d.queue_time_us, tablet-download-pool-de6c7753c8564107b858876fcfa5924d.queue_time_us) (tablet-download-pool-de6c7753c8564107b858876fcfa5924d.run_cpu_time_us, tablet-download-pool-de6c7753c8564107b858876fcfa5924d.run_cpu_time_us) (tablet-download-pool-de6c7753c8564107b858876fcfa5924d.run_wall_time_us, tablet-download-pool-de6c7753c8564107b858876fcfa5924d.run_wall_time_us) (tablet-download-pool-e7d5f02ccba34254a4d20951878b1214.queue_time_us, tablet-download-pool-e7d5f02ccba34254a4d20951878b1214.queue_time_us) (tablet-download-pool-e7d5f02ccba34254a4d20951878b1214.run_cpu_time_us, tablet-download-pool-e7d5f02ccba34254a4d20951878b1214.run_cpu_time_us) (tablet-download-pool-e7d5f02ccba34254a4d20951878b1214.run_wall_time_us, tablet-download-pool-e7d5f02ccba34254a4d20951878b1214.run_wall_time_us) (tablet-download-pool-fee16905387d428aa02cd2070c86c790.queue_time_us, tablet-download-pool-fee16905387d428aa02cd2070c86c790.queue_time_us) (tablet-download-pool-fee16905387d428aa02cd2070c86c790.run_cpu_time_us, tablet-download-pool-fee16905387d428aa02cd2070c86c790.run_cpu_time_us) (tablet-download-pool-fee16905387d428aa02cd2070c86c790.run_wall_time_us, tablet-download-pool-fee16905387d428aa02cd2070c86c790.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f27338a0dcd google::LogMessage::Fail() at ??:0
@ 0x7f27338a4b93 google::LogMessage::SendToLog() at ??:0
I20240501 14:05:43.020943 9910 raft_consensus.cc:2258] T 8cc5eec4600640f7a293877c825f8d96 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Raft consensus is shut down!
W20240501 14:05:43.027654 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:05:43.074869 5061 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.StartTabletCopy -> {remote=127.29.124.135:35329, user_credentials={real_user=slave}} blocked reactor thread for 47218.7us
I20240501 14:05:42.678107 5384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a726769d24284fedadd8e9e63cd27d05" candidate_uuid: "19a6d2d8a08142d5bffcde3a37234692" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: true dest_uuid: "be56773105e744919601127a94e98adf"
W20240501 14:05:43.075173 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:43.075273 5384 raft_consensus.cc:3037] T a726769d24284fedadd8e9e63cd27d05 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:05:43.077811 5384 raft_consensus.cc:2454] T a726769d24284fedadd8e9e63cd27d05 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 19a6d2d8a08142d5bffcde3a37234692 in term 2.
I20240501 14:05:42.774490 9604 raft_consensus.cc:1050] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf: attempting to promote NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 to VOTER
W20240501 14:05:43.078143 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:43.078858 9604 consensus_queue.cc:237] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } }
W20240501 14:05:42.798327 5321 connection.cc:535] client connection to 127.29.124.135:35329 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:05:43.080363 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:05:42.678301 7024 leader_election.cc:288] T a726769d24284fedadd8e9e63cd27d05 P 19a6d2d8a08142d5bffcde3a37234692 [CANDIDATE]: Term 2 election: Requested vote from peers e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377), 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129:44033), be56773105e744919601127a94e98adf (127.29.124.132:35459)
I20240501 14:05:43.018045 6959 leader_election.cc:302] T a726769d24284fedadd8e9e63cd27d05 P 19a6d2d8a08142d5bffcde3a37234692 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 04088ee3bea440a7bd5ff772152b9410, 19a6d2d8a08142d5bffcde3a37234692, e9c60449cf3c46f8aca4ab6c535302a4; no voters:
W20240501 14:05:43.081331 6959 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.29.124.129:44033, user_credentials={real_user=slave}} blocked reactor thread for 63332.4us
I20240501 14:05:43.082124 9528 raft_consensus.cc:2781] T a726769d24284fedadd8e9e63cd27d05 P 19a6d2d8a08142d5bffcde3a37234692 [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:05:43.083537 9910 ts_tablet_manager.cc:1875] T 8cc5eec4600640f7a293877c825f8d96 P e9c60449cf3c46f8aca4ab6c535302a4: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:05:43.088999 9899 tablet_copy_client.cc:523] T 1a7bbfc15a76450bbc9197037e856e08 P 6dac90f01e744f9c8701a404440a4c77: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:05:43.092579 9899 tablet_bootstrap.cc:492] T 1a7bbfc15a76450bbc9197037e856e08 P 6dac90f01e744f9c8701a404440a4c77: Bootstrap starting.
I20240501 14:05:43.069000 9742 ts_tablet_manager.cc:1367] T e7d5f02ccba34254a4d20951878b1214 P 938a24b0e6a040b7966225af26558ff8: Time spent bootstrapping tablet: real 0.538s user 0.050s sys 0.007s
I20240501 14:05:43.094529 9742 raft_consensus.cc:348] T e7d5f02ccba34254a4d20951878b1214 P 938a24b0e6a040b7966225af26558ff8 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }
I20240501 14:05:43.094802 9742 raft_consensus.cc:729] T e7d5f02ccba34254a4d20951878b1214 P 938a24b0e6a040b7966225af26558ff8 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 938a24b0e6a040b7966225af26558ff8, State: Initialized, Role: LEARNER
I20240501 14:05:43.095018 9742 consensus_queue.cc:260] T e7d5f02ccba34254a4d20951878b1214 P 938a24b0e6a040b7966225af26558ff8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 1.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: NON_VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: true } }
I20240501 14:05:43.095597 9742 ts_tablet_manager.cc:1398] T e7d5f02ccba34254a4d20951878b1214 P 938a24b0e6a040b7966225af26558ff8: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:05:43.096509 5406 tablet_copy_service.cc:342] P be56773105e744919601127a94e98adf: Request end of tablet copy session 938a24b0e6a040b7966225af26558ff8-e7d5f02ccba34254a4d20951878b1214 received from {username='slave'} at 127.29.124.133:50583
I20240501 14:05:43.096635 5406 tablet_copy_service.cc:433] P be56773105e744919601127a94e98adf: ending tablet copy session 938a24b0e6a040b7966225af26558ff8-e7d5f02ccba34254a4d20951878b1214 on tablet e7d5f02ccba34254a4d20951878b1214 with peer 938a24b0e6a040b7966225af26558ff8
I20240501 14:05:43.102875 7285 raft_consensus.cc:1261] T 2c39b53867e047faaf24cc6f87a8100c P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:05:43.103463 9616 consensus_queue.cc:1035] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
@ 0x7f27338a07cc google::LogMessage::Flush() at ??:0
@ 0x7f27338a1f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240501 14:05:43.117691 9528 raft_consensus.cc:686] T a726769d24284fedadd8e9e63cd27d05 P 19a6d2d8a08142d5bffcde3a37234692 [term 2 LEADER]: Becoming Leader. State: Replica: 19a6d2d8a08142d5bffcde3a37234692, State: Running, Role: LEADER
I20240501 14:05:43.118212 9528 consensus_queue.cc:237] T a726769d24284fedadd8e9e63cd27d05 P 19a6d2d8a08142d5bffcde3a37234692 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } }
I20240501 14:05:43.118510 9910 ts_tablet_manager.cc:1888] T 8cc5eec4600640f7a293877c825f8d96 P e9c60449cf3c46f8aca4ab6c535302a4: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.11
I20240501 14:05:43.118620 9910 log.cc:1196] T 8cc5eec4600640f7a293877c825f8d96 P e9c60449cf3c46f8aca4ab6c535302a4: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-1/wal/wals/8cc5eec4600640f7a293877c825f8d96
@ 0x7f27365e1742 kudu::TraceMetrics::InternName() at ??:0
@ 0x7f27365adf8b kudu::ThreadPool::ThreadPool() at ??:0
@ 0x7f27365a7cd6 kudu::ThreadPoolBuilder::Build() at ??:0
I20240501 14:05:43.124907 4854 catalog_manager.cc:5484] T a726769d24284fedadd8e9e63cd27d05 P 19a6d2d8a08142d5bffcde3a37234692 reported cstate change: term changed from 1 to 2, leader changed from 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129) to 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134). New cstate: current_term: 2 leader_uuid: "19a6d2d8a08142d5bffcde3a37234692" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
@ 0x7f27459fa8de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240501 14:05:43.141870 7024 raft_consensus.cc:1261] T 2c39b53867e047faaf24cc6f87a8100c P 19a6d2d8a08142d5bffcde3a37234692 [term 1 LEARNER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
@ 0x7f2745a12daf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240501 14:05:43.142788 6888 raft_consensus.cc:1261] T 2c39b53867e047faaf24cc6f87a8100c P 938a24b0e6a040b7966225af26558ff8 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:05:43.144553 4839 catalog_manager.cc:4830] TS e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377): tablet 8cc5eec4600640f7a293877c825f8d96 (table rebalance_test_table_6 [id=3c1513e944ee4b518373c4efa5e200cc]) successfully deleted
@ 0x7f2745b3d480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240501 14:05:43.149366 9816 consensus_queue.cc:1035] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
@ 0x7f2745b75a8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
@ 0x7f2745b58981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
W20240501 14:05:43.157310 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
@ 0x7f2745b5870e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
I20240501 14:05:43.165364 9816 consensus_queue.cc:1035] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
@ 0x7f27365b1b75 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f27365b519c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
W20240501 14:05:43.187592 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:43.194682 9616 raft_consensus.cc:2932] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:43.202199 7285 raft_consensus.cc:2932] T 2c39b53867e047faaf24cc6f87a8100c P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:43.202095 7024 raft_consensus.cc:2932] T 2c39b53867e047faaf24cc6f87a8100c P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) changed from NON_VOTER to VOTER. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:43.205413 4854 catalog_manager.cc:5484] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf reported cstate change: config changed from index 12 to 13, 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
@ 0x7f27365b5019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7f273659001e kudu::Thread::SuperviseThread() at ??:0
@ 0x7f27399156db start_thread at ??:0
@ 0x7f273275371f clone at ??:0
W20240501 14:05:43.225695 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
I20240501 14:05:43.227795 4994 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fee16905387d428aa02cd2070c86c790"
dest_uuid: "04088ee3bea440a7bd5ff772152b9410"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51304
I20240501 14:05:43.228058 4994 raft_consensus.cc:595] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:05:43.378960 5255 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968"
dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:45442
I20240501 14:05:43.379252 5255 raft_consensus.cc:595] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEADER]: Received request to transfer leadership
I20240501 14:05:43.386709 9764 raft_consensus.cc:979] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c: : Instructing follower be56773105e744919601127a94e98adf to start an election
I20240501 14:05:43.386852 9764 raft_consensus.cc:1067] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEADER]: Signalling peer be56773105e744919601127a94e98adf to start an election
I20240501 14:05:43.687306 7004 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
W20240501 14:05:43.594249 5191 connection.cc:535] client connection to 127.29.124.133:35635 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:05:43.593561 5061 connection.cc:535] client connection to 127.29.124.133:35635 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:05:43.471880 9817 raft_consensus.cc:979] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410: : Instructing follower e9c60449cf3c46f8aca4ab6c535302a4 to start an election
I20240501 14:05:43.745039 9817 raft_consensus.cc:1067] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEADER]: Signalling peer e9c60449cf3c46f8aca4ab6c535302a4 to start an election
I20240501 14:05:43.666916 5236 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:43.736271 5126 raft_consensus.cc:1261] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 [term 2 FOLLOWER]: Refusing update from remote peer 19a6d2d8a08142d5bffcde3a37234692: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240501 14:05:43.398782 9899 tablet_bootstrap.cc:492] T 1a7bbfc15a76450bbc9197037e856e08 P 6dac90f01e744f9c8701a404440a4c77: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:05:43.701570 7267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:43.745836 9899 tablet_bootstrap.cc:492] T 1a7bbfc15a76450bbc9197037e856e08 P 6dac90f01e744f9c8701a404440a4c77: Bootstrap complete.
I20240501 14:05:43.746151 9899 ts_tablet_manager.cc:1367] T 1a7bbfc15a76450bbc9197037e856e08 P 6dac90f01e744f9c8701a404440a4c77: Time spent bootstrapping tablet: real 0.654s user 0.053s sys 0.003s
I20240501 14:05:43.746984 9899 raft_consensus.cc:348] T 1a7bbfc15a76450bbc9197037e856e08 P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false replace: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } }
I20240501 14:05:43.747294 9899 raft_consensus.cc:729] T 1a7bbfc15a76450bbc9197037e856e08 P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6dac90f01e744f9c8701a404440a4c77, State: Initialized, Role: LEARNER
I20240501 14:05:43.747488 9899 consensus_queue.cc:260] T 1a7bbfc15a76450bbc9197037e856e08 P 6dac90f01e744f9c8701a404440a4c77 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false replace: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } }
I20240501 14:05:43.388636 5384 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968"
dest_uuid: "be56773105e744919601127a94e98adf"
from {username='slave'} at 127.29.124.131:37793
I20240501 14:05:43.747967 5384 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:05:43.748113 5384 raft_consensus.cc:3037] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:05:43.595265 5321 connection.cc:535] client connection to 127.29.124.133:35635 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:05:43.750866 5384 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:05:43.751003 9528 consensus_queue.cc:1035] T a726769d24284fedadd8e9e63cd27d05 P 19a6d2d8a08142d5bffcde3a37234692 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:05:43.751842 5384 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 2 election: Requested vote from peers 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:05:43.474720 4994 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fee16905387d428aa02cd2070c86c790"
dest_uuid: "04088ee3bea440a7bd5ff772152b9410"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:51334
I20240501 14:05:43.752349 4994 raft_consensus.cc:595] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEADER]: Received request to transfer leadership
I20240501 14:05:43.752852 5255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c"
I20240501 14:05:43.753083 5255 raft_consensus.cc:3032] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:05:43.753192 5255 raft_consensus.cc:729] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 1c3fdc5272484ac09ef3e5e36a08f27c, State: Running, Role: LEADER
I20240501 14:05:43.707397 5365 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:43.733843 5386 raft_consensus.cc:1261] T a726769d24284fedadd8e9e63cd27d05 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Refusing update from remote peer 19a6d2d8a08142d5bffcde3a37234692: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
W20240501 14:05:43.594029 4930 connection.cc:535] client connection to 127.29.124.133:35635 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240501 14:05:43.753525 5255 consensus_queue.cc:260] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:05:43.753909 5255 raft_consensus.cc:3037] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:05:43.754856 5125 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fee16905387d428aa02cd2070c86c790"
dest_uuid: "e9c60449cf3c46f8aca4ab6c535302a4"
from {username='slave'} at 127.29.124.129:40609
I20240501 14:05:43.755123 5125 raft_consensus.cc:482] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:05:43.755234 5125 raft_consensus.cc:3037] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:05:43.756620 5255 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 2.
I20240501 14:05:43.666533 4973 tablet_service.cc:1429] Tablet server has 5 leaders and 0 scanners
I20240501 14:05:43.757670 9528 consensus_queue.cc:1035] T a726769d24284fedadd8e9e63cd27d05 P 19a6d2d8a08142d5bffcde3a37234692 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:05:43.781394 5125 raft_consensus.cc:504] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } }
I20240501 14:05:43.782706 4994 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fee16905387d428aa02cd2070c86c790" candidate_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: true dest_uuid: "04088ee3bea440a7bd5ff772152b9410"
I20240501 14:05:43.782912 4994 raft_consensus.cc:3032] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:05:43.782980 4994 raft_consensus.cc:729] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 04088ee3bea440a7bd5ff772152b9410, State: Running, Role: LEADER
I20240501 14:05:43.783268 4994 consensus_queue.cc:260] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } }
I20240501 14:05:43.783654 4994 raft_consensus.cc:3037] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:05:43.785858 5125 leader_election.cc:288] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 [CANDIDATE]: Term 2 election: Requested vote from peers 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129:44033), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665)
I20240501 14:05:43.787751 4994 raft_consensus.cc:2454] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9c60449cf3c46f8aca4ab6c535302a4 in term 2.
I20240501 14:05:43.822888 9899 ts_tablet_manager.cc:1398] T 1a7bbfc15a76450bbc9197037e856e08 P 6dac90f01e744f9c8701a404440a4c77: Time spent starting tablet: real 0.077s user 0.008s sys 0.004s
I20240501 14:05:43.825639 5276 tablet_copy_service.cc:342] P 1c3fdc5272484ac09ef3e5e36a08f27c: Request end of tablet copy session 6dac90f01e744f9c8701a404440a4c77-1a7bbfc15a76450bbc9197037e856e08 received from {username='slave'} at 127.29.124.136:49141
I20240501 14:05:43.825758 5276 tablet_copy_service.cc:433] P 1c3fdc5272484ac09ef3e5e36a08f27c: ending tablet copy session 6dac90f01e744f9c8701a404440a4c77-1a7bbfc15a76450bbc9197037e856e08 on tablet 1a7bbfc15a76450bbc9197037e856e08 with peer 6dac90f01e744f9c8701a404440a4c77
W20240501 14:05:43.828516 4930 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.124.133:35635, user_credentials={real_user=slave}} blocked reactor thread for 74755.9us
W20240501 14:05:43.830426 5061 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.124.134:40665, user_credentials={real_user=slave}} blocked reactor thread for 73469.6us
W20240501 14:05:43.831774 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:43.833796 7285 raft_consensus.cc:1203] T 1a7bbfc15a76450bbc9197037e856e08 P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
I20240501 14:05:43.838979 7024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fee16905387d428aa02cd2070c86c790" candidate_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: true dest_uuid: "19a6d2d8a08142d5bffcde3a37234692"
I20240501 14:05:43.839304 7024 raft_consensus.cc:3037] T fee16905387d428aa02cd2070c86c790 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:05:43.840917 4994 raft_consensus.cc:1261] T a726769d24284fedadd8e9e63cd27d05 P 04088ee3bea440a7bd5ff772152b9410 [term 2 FOLLOWER]: Refusing update from remote peer 19a6d2d8a08142d5bffcde3a37234692: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
W20240501 14:05:43.841364 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:43.844306 7024 raft_consensus.cc:2454] T fee16905387d428aa02cd2070c86c790 P 19a6d2d8a08142d5bffcde3a37234692 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9c60449cf3c46f8aca4ab6c535302a4 in term 2.
I20240501 14:05:43.850478 5061 leader_election.cc:302] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 04088ee3bea440a7bd5ff772152b9410, 19a6d2d8a08142d5bffcde3a37234692, e9c60449cf3c46f8aca4ab6c535302a4; no voters:
I20240501 14:05:43.850945 9520 raft_consensus.cc:2781] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 [term 2 FOLLOWER]: Leader election won for term 2
W20240501 14:05:43.851064 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:43.851097 9520 raft_consensus.cc:686] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 [term 2 LEADER]: Becoming Leader. State: Replica: e9c60449cf3c46f8aca4ab6c535302a4, State: Running, Role: LEADER
I20240501 14:05:43.851389 9520 consensus_queue.cc:237] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } }
W20240501 14:05:43.858865 5321 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.124.133:35635, user_credentials={real_user=slave}} blocked reactor thread for 109191us
I20240501 14:05:43.870555 9528 consensus_queue.cc:1035] T a726769d24284fedadd8e9e63cd27d05 P 19a6d2d8a08142d5bffcde3a37234692 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 14:05:43.876121 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:05:43.876353 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:05:43.877094 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:43.898715 7024 consensus_queue.cc:237] T a726769d24284fedadd8e9e63cd27d05 P 19a6d2d8a08142d5bffcde3a37234692 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } }
I20240501 14:05:43.903597 5126 raft_consensus.cc:1261] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 [term 2 FOLLOWER]: Refusing update from remote peer 19a6d2d8a08142d5bffcde3a37234692: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 14:05:43.904484 10164 consensus_queue.cc:1035] T a726769d24284fedadd8e9e63cd27d05 P 19a6d2d8a08142d5bffcde3a37234692 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240501 14:05:43.910399 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:05:43.910836 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:05:43.911183 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:05:43.911332 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:05:43.912894 5061 leader_election.cc:334] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
I20240501 14:05:43.914623 4847 catalog_manager.cc:5484] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 reported cstate change: term changed from 1 to 2, leader changed from 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129) to e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130). New cstate: current_term: 2 leader_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240501 14:05:43.916564 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:43.920248 5106 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:05:43.961094 10164 raft_consensus.cc:2932] T a726769d24284fedadd8e9e63cd27d05 P 19a6d2d8a08142d5bffcde3a37234692 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 7 to 9, VOTER 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:43.965607 4840 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a726769d24284fedadd8e9e63cd27d05 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:05:43.965960 5126 raft_consensus.cc:2932] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 7 to 9, VOTER 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:43.967746 4853 catalog_manager.cc:5484] T a726769d24284fedadd8e9e63cd27d05 P 19a6d2d8a08142d5bffcde3a37234692 reported cstate change: config changed from index 7 to 9, VOTER 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129) evicted. New cstate: current_term: 2 leader_uuid: "19a6d2d8a08142d5bffcde3a37234692" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 14:05:43.977069 5321 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:05:43.978402 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:05:43.978896 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:05:43.979498 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
I20240501 14:05:43.980194 5321 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:05:43.981372 10169 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240501 14:05:43.984638 5384 raft_consensus.cc:1261] T a726769d24284fedadd8e9e63cd27d05 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Refusing update from remote peer 19a6d2d8a08142d5bffcde3a37234692: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 14:05:43.985605 10164 consensus_queue.cc:1035] T a726769d24284fedadd8e9e63cd27d05 P 19a6d2d8a08142d5bffcde3a37234692 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:05:43.987210 5384 raft_consensus.cc:2932] T a726769d24284fedadd8e9e63cd27d05 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 7 to 9, VOTER 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:05:44.023008 4973 tablet_service.cc:1514] Processing DeleteTablet for tablet a726769d24284fedadd8e9e63cd27d05 with delete_type TABLET_DATA_TOMBSTONED (TS 04088ee3bea440a7bd5ff772152b9410 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:41194
I20240501 14:05:44.032474 10195 tablet_replica.cc:330] T a726769d24284fedadd8e9e63cd27d05 P 04088ee3bea440a7bd5ff772152b9410: stopping tablet replica
I20240501 14:05:44.032711 10195 raft_consensus.cc:2229] T a726769d24284fedadd8e9e63cd27d05 P 04088ee3bea440a7bd5ff772152b9410 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240501 14:05:44.034986 10195 raft_consensus.cc:2258] T a726769d24284fedadd8e9e63cd27d05 P 04088ee3bea440a7bd5ff772152b9410 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240501 14:05:44.037614 10195 ts_tablet_manager.cc:1875] T a726769d24284fedadd8e9e63cd27d05 P 04088ee3bea440a7bd5ff772152b9410: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240501 14:05:44.044008 10195 ts_tablet_manager.cc:1888] T a726769d24284fedadd8e9e63cd27d05 P 04088ee3bea440a7bd5ff772152b9410: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.8
I20240501 14:05:44.044231 10195 log.cc:1196] T a726769d24284fedadd8e9e63cd27d05 P 04088ee3bea440a7bd5ff772152b9410: Deleting WAL directory at /tmp/dist-test-taskUv7aek/test-tmp/rebalancer_tool-test.3.ConcurrentRebalancersTest.TwoConcurrentRebalancers_1.1714572223945197-30194-0/raft_consensus-itest-cluster/ts-0/wal/wals/a726769d24284fedadd8e9e63cd27d05
I20240501 14:05:44.045187 4839 catalog_manager.cc:4830] TS 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129:44033): tablet a726769d24284fedadd8e9e63cd27d05 (table rebalance_test_table_2 [id=facc024bfd644356a8879fd9661c928d]) successfully deleted
I20240501 14:05:44.172320 5386 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "420642af520d45ffbc7f2d35f93e75a7"
dest_uuid: "be56773105e744919601127a94e98adf"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:55206
I20240501 14:05:44.172600 5386 raft_consensus.cc:595] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf [term 1 LEADER]: Received request to transfer leadership
I20240501 14:05:44.234779 4994 raft_consensus.cc:1261] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [term 2 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240501 14:05:44.235602 9690 consensus_queue.cc:1035] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20240501 14:05:44.248121 4840 catalog_manager.cc:4573] TS 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 8cc5eec4600640f7a293877c825f8d96: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:05:44.260627 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:05:44.286410 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:05:44.309808 7024 raft_consensus.cc:1261] T fee16905387d428aa02cd2070c86c790 P 19a6d2d8a08142d5bffcde3a37234692 [term 2 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 13. (index mismatch)
I20240501 14:05:44.310559 9690 consensus_queue.cc:1035] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20240501 14:05:44.328063 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:44.328135 7338 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskUv7aek/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240501 14:05:26.847333 7343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:05:26.855031 7343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:05:26.867131 7343 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:05:29.752857 7343 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240501 14:05:31.507501 7343 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 41272b857ab045629ca60ab2467097b0: could not find any suitable replica to move from server 1c3fdc5272484ac09ef3e5e36a08f27c to server 6dac90f01e744f9c8701a404440a4c77
W20240501 14:05:31.508159 7343 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3c1513e944ee4b518373c4efa5e200cc: could not find any suitable replica to move from server e9c60449cf3c46f8aca4ab6c535302a4 to server 19a6d2d8a08142d5bffcde3a37234692
W20240501 14:05:31.508455 7343 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3794b1ccb5004f7cb92a229ef6ce3595: could not find any suitable replica to move from server e9c60449cf3c46f8aca4ab6c535302a4 to server 938a24b0e6a040b7966225af26558ff8
W20240501 14:05:31.508579 7343 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 403fdb578f8e4066a56f685925e83936: could not find any suitable replica to move from server 1c3fdc5272484ac09ef3e5e36a08f27c to server 6dac90f01e744f9c8701a404440a4c77
W20240501 14:05:31.508708 7343 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 84aeba643ddd44cfbbd6a34f81703482: could not find any suitable replica to move from server 04088ee3bea440a7bd5ff772152b9410 to server 938a24b0e6a040b7966225af26558ff8
W20240501 14:05:31.510622 7343 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3c1513e944ee4b518373c4efa5e200cc: could not find any suitable replica to move from server be56773105e744919601127a94e98adf to server 6dac90f01e744f9c8701a404440a4c77
W20240501 14:05:31.510752 7343 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table facc024bfd644356a8879fd9661c928d: could not find any suitable replica to move from server e9c60449cf3c46f8aca4ab6c535302a4 to server 4360b59e3a154688a79ca5f9c0a37bc9
W20240501 14:05:31.511001 7343 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5fc33bdfba464f65b68de5887d425ab7: could not find any suitable replica to move from server e9c60449cf3c46f8aca4ab6c535302a4 to server 6dac90f01e744f9c8701a404440a4c77
W20240501 14:05:31.511149 7343 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f26ba2f4e5654a5fa72021df5929200d: could not find any suitable replica to move from server 04088ee3bea440a7bd5ff772152b9410 to server 19a6d2d8a08142d5bffcde3a37234692
W20240501 14:05:31.511278 7343 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7af67fe405db4bffbc53cb288b8bb715: could not find any suitable replica to move from server 04088ee3bea440a7bd5ff772152b9410 to server 19a6d2d8a08142d5bffcde3a37234692
W20240501 14:05:31.511392 7343 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 763c884aecd640b49c65c692fbde9422: could not find any suitable replica to move from server 1c3fdc5272484ac09ef3e5e36a08f27c to server 6dac90f01e744f9c8701a404440a4c77
W20240501 14:05:31.511560 7343 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3794b1ccb5004f7cb92a229ef6ce3595: could not find any suitable replica to move from server be56773105e744919601127a94e98adf to server 4360b59e3a154688a79ca5f9c0a37bc9
W20240501 14:05:31.511682 7343 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5fc33bdfba464f65b68de5887d425ab7: could not find any suitable replica to move from server 1c3fdc5272484ac09ef3e5e36a08f27c to server 19a6d2d8a08142d5bffcde3a37234692
W20240501 14:05:31.512519 7343 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3c1513e944ee4b518373c4efa5e200cc: could not find any suitable replica to move from server 04088ee3bea440a7bd5ff772152b9410 to server 938a24b0e6a040b7966225af26558ff8
W20240501 14:05:31.512636 7343 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 403fdb578f8e4066a56f685925e83936: could not find any suitable replica to move from server e9c60449cf3c46f8aca4ab6c535302a4 to server 6dac90f01e744f9c8701a404440a4c77
I20240501 14:05:31.899919 7343 rebalancer_tool.cc:1074] tablet ac4b234f794149e4b6def7a3b358942d: '04088ee3bea440a7bd5ff772152b9410' -> '4360b59e3a154688a79ca5f9c0a37bc9' move scheduled
I20240501 14:05:32.124130 7343 rebalancer_tool.cc:1074] tablet fee16905387d428aa02cd2070c86c790: 'be56773105e744919601127a94e98adf' -> '938a24b0e6a040b7966225af26558ff8' move scheduled
I20240501 14:05:32.624787 7343 rebalancer_tool.cc:1074] tablet f9cb784f60774ea7b6e70ccfebf867a8: '1c3fdc5272484ac09ef3e5e36a08f27c' -> '6dac90f01e744f9c8701a404440a4c77' move scheduled
I20240501 14:05:32.958925 7343 rebalancer_tool.cc:1074] tablet f4d2b8c005b64bbb8e6621da068d15e5: 'e9c60449cf3c46f8aca4ab6c535302a4' -> '19a6d2d8a08142d5bffcde3a37234692' move scheduled
I20240501 14:05:33.198226 7343 rebalancer_tool.cc:1074] tablet 9b88c43cb8704583a3e65c5c35c205f6: '04088ee3bea440a7bd5ff772152b9410' -> '4360b59e3a154688a79ca5f9c0a37bc9' move scheduled
I20240501 14:05:33.681530 7343 rebalancer_tool.cc:1074] tablet 8cc5eec4600640f7a293877c825f8d96: 'be56773105e744919601127a94e98adf' -> '938a24b0e6a040b7966225af26558ff8' move scheduled
I20240501 14:05:34.083288 7343 rebalancer_tool.cc:1074] tablet 2c39b53867e047faaf24cc6f87a8100c: '1c3fdc5272484ac09ef3e5e36a08f27c' -> '19a6d2d8a08142d5bffcde3a37234692' move scheduled
I20240501 14:05:34.640336 7343 rebalancer_tool.cc:1074] tablet e7d5f02ccba34254a4d20951878b1214: '04088ee3bea440a7bd5ff772152b9410' -> '6dac90f01e744f9c8701a404440a4c77' move scheduled
I20240501 14:05:34.883245 7343 rebalancer_tool.cc:1074] tablet ec1e81011c6a46bc8a39aecf3a5e71e4: 'e9c60449cf3c46f8aca4ab6c535302a4' -> '4360b59e3a154688a79ca5f9c0a37bc9' move scheduled
I20240501 14:05:35.122925 7343 rebalancer_tool.cc:1074] tablet 61a274b24f53458f88c2cde4f8dbe4f4: 'be56773105e744919601127a94e98adf' -> '938a24b0e6a040b7966225af26558ff8' move scheduled
I20240501 14:05:35.429147 7343 rebalancer_tool.cc:1084] tablet 9afe3b20a43d4b5bb800af39f945948d: '1c3fdc5272484ac09ef3e5e36a08f27c' -> '19a6d2d8a08142d5bffcde3a37234692' move ignored: Not found: Peer with uuid 1c3fdc5272484ac09ef3e5e36a08f27c not found in consensus config
I20240501 14:05:35.429409 7343 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 14:05:38.167178 7343 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 403fdb578f8e4066a56f685925e83936: could not find any suitable replica to move from server 1c3fdc5272484ac09ef3e5e36a08f27c to server 4360b59e3a154688a79ca5f9c0a37bc9
W20240501 14:05:38.167426 7343 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3c1513e944ee4b518373c4efa5e200cc: could not find any suitable replica to move from server be56773105e744919601127a94e98adf to server 4360b59e3a154688a79ca5f9c0a37bc9
W20240501 14:05:38.167532 7343 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3794b1ccb5004f7cb92a229ef6ce3595: could not find any suitable replica to move from server e9c60449cf3c46f8aca4ab6c535302a4 to server 19a6d2d8a08142d5bffcde3a37234692
I20240501 14:05:38.387912 7343 rebalancer_tool.cc:1074] tablet 51cbcf4888c440c2ad096c988ab2d968: '1c3fdc5272484ac09ef3e5e36a08f27c' -> '' move scheduled
I20240501 14:05:38.507962 7343 rebalancer_tool.cc:1074] tablet 43b46fcf827c4275a47291afb107c290: 'e9c60449cf3c46f8aca4ab6c535302a4' -> '6dac90f01e744f9c8701a404440a4c77' move scheduled
I20240501 14:05:38.785037 7343 rebalancer_tool.cc:1074] tablet 3d6859d53fd747a5a5e94722626fa522: '1c3fdc5272484ac09ef3e5e36a08f27c' -> '19a6d2d8a08142d5bffcde3a37234692' move scheduled
I20240501 14:05:38.981599 7343 rebalancer_tool.cc:1074] tablet 88ed5438fc934adc9fb98c6cb4476598: '04088ee3bea440a7bd5ff772152b9410' -> '4360b59e3a154688a79ca5f9c0a37bc9' move scheduled
I20240501 14:05:39.475939 7343 rebalancer_tool.cc:1074] tablet b284aa82594a44f5885f32fa05844913: 'be56773105e744919601127a94e98adf' -> '938a24b0e6a040b7966225af26558ff8' move scheduled
I20240501 14:05:39.730691 7343 rebalancer_tool.cc:1084] tablet a726769d24284fedadd8e9e63cd27d05: 'e9c60449cf3c46f8aca4ab6c535302a4' -> '19a6d2d8a08142d5bffcde3a37234692' move ignored: Illegal state: tablet a726769d24284fedadd8e9e63cd27d05: replica 19a6d2d8a08142d5bffcde3a37234692 is already present
I20240501 14:05:39.730808 7343 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 14:05:42.745445 7343 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f26ba2f4e5654a5fa72021df5929200d: could not find any suitable replica to move from server e9c60449cf3c46f8aca4ab6c535302a4 to server 1c3fdc5272484ac09ef3e5e36a08f27c
W20240501 14:05:42.747426 7343 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7af67fe405db4bffbc53cb288b8bb715: could not find any suitable replica to move from server 4360b59e3a154688a79ca5f9c0a37bc9 to server 04088ee3bea440a7bd5ff772152b9410
W20240501 14:05:42.753635 7343 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 84aeba643ddd44cfbbd6a34f81703482: could not find any suitable replica to move from server 04088ee3bea440a7bd5ff772152b9410 to server 1c3fdc5272484ac09ef3e5e36a08f27c
I20240501 14:05:42.841176 7343 rebalancer_tool.cc:1084] tablet 1a7bbfc15a76450bbc9197037e856e08: '19a6d2d8a08142d5bffcde3a37234692' -> '6dac90f01e744f9c8701a404440a4c77' move ignored: Illegal state: tablet 1a7bbfc15a76450bbc9197037e856e08: replica 6dac90f01e744f9c8701a404440a4c77 is already present
I20240501 14:05:42.841382 7343 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): unacceptable health status UNAVAILABLE
I20240501 14:05:44.367218 4973 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:44.381035 9616 raft_consensus.cc:979] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf: : Instructing follower 1c3fdc5272484ac09ef3e5e36a08f27c to start an election
I20240501 14:05:44.381203 9616 raft_consensus.cc:1067] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf [term 1 LEADER]: Signalling peer 1c3fdc5272484ac09ef3e5e36a08f27c to start an election
I20240501 14:05:44.382162 5254 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "420642af520d45ffbc7f2d35f93e75a7"
dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c"
from {username='slave'} at 127.29.124.132:53607
I20240501 14:05:44.382417 5254 raft_consensus.cc:482] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240501 14:05:44.382531 5254 raft_consensus.cc:3037] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:05:44.386854 5254 raft_consensus.cc:504] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } }
I20240501 14:05:44.388178 5386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420642af520d45ffbc7f2d35f93e75a7" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "be56773105e744919601127a94e98adf"
I20240501 14:05:44.388409 5386 raft_consensus.cc:3032] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf [term 1 LEADER]: Stepping down as leader of term 1
I20240501 14:05:44.388490 5386 raft_consensus.cc:729] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf [term 1 LEADER]: Becoming Follower/Learner. State: Replica: be56773105e744919601127a94e98adf, State: Running, Role: LEADER
I20240501 14:05:44.388818 5386 consensus_queue.cc:260] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } }
I20240501 14:05:44.389286 5386 raft_consensus.cc:3037] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Advancing to term 2
W20240501 14:05:44.389426 5191 leader_election.cc:334] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:05:44.390455 5254 leader_election.cc:288] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 2 election: Requested vote from peers be56773105e744919601127a94e98adf (127.29.124.132:35459), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635), 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377)
I20240501 14:05:44.392096 5386 raft_consensus.cc:2454] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 2.
I20240501 14:05:44.394662 7285 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "420642af520d45ffbc7f2d35f93e75a7" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "6dac90f01e744f9c8701a404440a4c77"
I20240501 14:05:44.395020 7285 raft_consensus.cc:3037] T 420642af520d45ffbc7f2d35f93e75a7 P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Advancing to term 2
I20240501 14:05:44.398502 7285 raft_consensus.cc:2454] T 420642af520d45ffbc7f2d35f93e75a7 P 6dac90f01e744f9c8701a404440a4c77 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 2.
I20240501 14:05:44.399142 5192 leader_election.cc:302] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 4 responses out of 4 voters: 3 yes votes; 1 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, 6dac90f01e744f9c8701a404440a4c77, be56773105e744919601127a94e98adf; no voters: 938a24b0e6a040b7966225af26558ff8
I20240501 14:05:44.399713 9765 raft_consensus.cc:2781] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader election won for term 2
I20240501 14:05:44.399852 9765 raft_consensus.cc:686] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 LEADER]: Becoming Leader. State: Replica: 1c3fdc5272484ac09ef3e5e36a08f27c, State: Running, Role: LEADER
I20240501 14:05:44.400177 9765 consensus_queue.cc:237] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } }
I20240501 14:05:44.418879 4847 catalog_manager.cc:5484] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c reported cstate change: term changed from 1 to 2, leader changed from be56773105e744919601127a94e98adf (127.29.124.132) to 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131). New cstate: current_term: 2 leader_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:05:44.442919 9764 raft_consensus.cc:1050] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c: attempting to promote NON_VOTER 6dac90f01e744f9c8701a404440a4c77 to VOTER
I20240501 14:05:44.444144 9764 consensus_queue.cc:237] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false replace: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } }
I20240501 14:05:44.485100 7004 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:44.495607 5236 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:44.506572 7285 raft_consensus.cc:1261] T 1a7bbfc15a76450bbc9197037e856e08 P 6dac90f01e744f9c8701a404440a4c77 [term 1 LEARNER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:05:44.507763 7024 raft_consensus.cc:1261] T 1a7bbfc15a76450bbc9197037e856e08 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:05:44.508410 10309 consensus_queue.cc:1035] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:05:44.509128 10309 consensus_queue.cc:1035] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240501 14:05:44.546612 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:44.567977 10309 raft_consensus.cc:2932] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false replace: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:44.569401 7267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:44.574136 7024 raft_consensus.cc:2932] T 1a7bbfc15a76450bbc9197037e856e08 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false replace: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:44.574720 4853 catalog_manager.cc:5484] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c reported cstate change: config changed from index 10 to 11, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:05:44.582930 5254 consensus_queue.cc:237] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } }
I20240501 14:05:44.586126 7287 raft_consensus.cc:1261] T 1a7bbfc15a76450bbc9197037e856e08 P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:05:44.586699 10309 consensus_queue.cc:1035] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:05:44.587941 7287 raft_consensus.cc:2932] T 1a7bbfc15a76450bbc9197037e856e08 P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false replace: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:44.591506 5365 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:05:44.591660 5106 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240501 14:05:44.592203 7024 raft_consensus.cc:1261] T 1a7bbfc15a76450bbc9197037e856e08 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:05:44.595331 9764 consensus_queue.cc:1035] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:05:44.597671 10309 raft_consensus.cc:2932] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:44.601692 4838 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1a7bbfc15a76450bbc9197037e856e08 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:05:44.610639 4853 catalog_manager.cc:5484] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c reported cstate change: config changed from index 11 to 12, VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) evicted. New cstate: current_term: 1 leader_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:05:44.613781 7024 raft_consensus.cc:2932] T 1a7bbfc15a76450bbc9197037e856e08 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:05:44.632190 7286 raft_consensus.cc:2932] T 1a7bbfc15a76450bbc9197037e856e08 P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
W20240501 14:05:44.712339 6959 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.29.124.132:35459, user_credentials={real_user=slave}} blocked reactor thread for 56093.1us
I20240501 14:05:44.717078 4973 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240501 14:05:44.718770 7004 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240501 14:05:44.736856 5236 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240501 14:05:44.759716 7267 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240501 14:05:44.783866 5365 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240501 14:05:44.804137 5106 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
W20240501 14:05:44.806473 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
I20240501 14:05:44.807818 5386 raft_consensus.cc:1261] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 14:05:44.808570 9764 consensus_queue.cc:1035] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240501 14:05:44.879663 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:05:44.879797 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:05:45.025183 7285 raft_consensus.cc:1261] T 420642af520d45ffbc7f2d35f93e75a7 P 6dac90f01e744f9c8701a404440a4c77 [term 2 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240501 14:05:45.025817 10309 consensus_queue.cc:1035] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:05:45.065845 7339 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskUv7aek/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240501 14:05:26.839151 7345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:05:26.839675 7345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:05:26.853619 7345 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:05:29.345430 7345 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240501 14:05:30.958381 7345 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 41272b857ab045629ca60ab2467097b0: could not find any suitable replica to move from server 1c3fdc5272484ac09ef3e5e36a08f27c to server 4360b59e3a154688a79ca5f9c0a37bc9
W20240501 14:05:30.958636 7345 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3c1513e944ee4b518373c4efa5e200cc: could not find any suitable replica to move from server e9c60449cf3c46f8aca4ab6c535302a4 to server 4360b59e3a154688a79ca5f9c0a37bc9
W20240501 14:05:30.959156 7345 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5fc33bdfba464f65b68de5887d425ab7: could not find any suitable replica to move from server be56773105e744919601127a94e98adf to server 19a6d2d8a08142d5bffcde3a37234692
W20240501 14:05:30.959606 7345 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 84aeba643ddd44cfbbd6a34f81703482: could not find any suitable replica to move from server 04088ee3bea440a7bd5ff772152b9410 to server 6dac90f01e744f9c8701a404440a4c77
W20240501 14:05:30.959771 7345 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 763c884aecd640b49c65c692fbde9422: could not find any suitable replica to move from server e9c60449cf3c46f8aca4ab6c535302a4 to server 938a24b0e6a040b7966225af26558ff8
W20240501 14:05:30.960096 7345 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table facc024bfd644356a8879fd9661c928d: could not find any suitable replica to move from server 04088ee3bea440a7bd5ff772152b9410 to server 6dac90f01e744f9c8701a404440a4c77
W20240501 14:05:30.960256 7345 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3794b1ccb5004f7cb92a229ef6ce3595: could not find any suitable replica to move from server e9c60449cf3c46f8aca4ab6c535302a4 to server 19a6d2d8a08142d5bffcde3a37234692
W20240501 14:05:30.960428 7345 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 403fdb578f8e4066a56f685925e83936: could not find any suitable replica to move from server 1c3fdc5272484ac09ef3e5e36a08f27c to server 6dac90f01e744f9c8701a404440a4c77
W20240501 14:05:30.960539 7345 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5fc33bdfba464f65b68de5887d425ab7: could not find any suitable replica to move from server 1c3fdc5272484ac09ef3e5e36a08f27c to server 4360b59e3a154688a79ca5f9c0a37bc9
W20240501 14:05:30.960698 7345 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7af67fe405db4bffbc53cb288b8bb715: could not find any suitable replica to move from server be56773105e744919601127a94e98adf to server 4360b59e3a154688a79ca5f9c0a37bc9
W20240501 14:05:30.960825 7345 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f26ba2f4e5654a5fa72021df5929200d: could not find any suitable replica to move from server be56773105e744919601127a94e98adf to server 19a6d2d8a08142d5bffcde3a37234692
W20240501 14:05:30.960999 7345 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table facc024bfd644356a8879fd9661c928d: could not find any suitable replica to move from server e9c60449cf3c46f8aca4ab6c535302a4 to server 4360b59e3a154688a79ca5f9c0a37bc9
W20240501 14:05:30.961103 7345 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 84aeba643ddd44cfbbd6a34f81703482: could not find any suitable replica to move from server 1c3fdc5272484ac09ef3e5e36a08f27c to server 6dac90f01e744f9c8701a404440a4c77
W20240501 14:05:30.961231 7345 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3c1513e944ee4b518373c4efa5e200cc: could not find any suitable replica to move from server 04088ee3bea440a7bd5ff772152b9410 to server 938a24b0e6a040b7966225af26558ff8
W20240501 14:05:30.961409 7345 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3794b1ccb5004f7cb92a229ef6ce3595: could not find any suitable replica to move from server be56773105e744919601127a94e98adf to server 19a6d2d8a08142d5bffcde3a37234692
I20240501 14:05:31.158522 7345 rebalancer_tool.cc:1074] tablet de6c7753c8564107b858876fcfa5924d: '04088ee3bea440a7bd5ff772152b9410' -> '6dac90f01e744f9c8701a404440a4c77' move scheduled
I20240501 14:05:31.437739 7345 rebalancer_tool.cc:1074] tablet fee16905387d428aa02cd2070c86c790: 'be56773105e744919601127a94e98adf' -> '4360b59e3a154688a79ca5f9c0a37bc9' move scheduled
I20240501 14:05:31.568876 7345 rebalancer_tool.cc:1074] tablet 1a7bbfc15a76450bbc9197037e856e08: 'e9c60449cf3c46f8aca4ab6c535302a4' -> '19a6d2d8a08142d5bffcde3a37234692' move scheduled
I20240501 14:05:31.687326 7345 rebalancer_tool.cc:1074] tablet 2c39b53867e047faaf24cc6f87a8100c: '1c3fdc5272484ac09ef3e5e36a08f27c' -> '938a24b0e6a040b7966225af26558ff8' move scheduled
I20240501 14:05:31.869091 7345 rebalancer_tool.cc:1074] tablet 9d32e0707d244f2c9bbcb57f62b09f31: '04088ee3bea440a7bd5ff772152b9410' -> '6dac90f01e744f9c8701a404440a4c77' move scheduled
I20240501 14:05:32.241549 7345 rebalancer_tool.cc:1074] tablet 8cc5eec4600640f7a293877c825f8d96: 'be56773105e744919601127a94e98adf' -> '19a6d2d8a08142d5bffcde3a37234692' move scheduled
I20240501 14:05:32.652869 7345 rebalancer_tool.cc:1074] tablet 9fd94635797d447cbdc9181ee91f465b: 'e9c60449cf3c46f8aca4ab6c535302a4' -> '4360b59e3a154688a79ca5f9c0a37bc9' move scheduled
I20240501 14:05:32.894654 7345 rebalancer_tool.cc:1074] tablet 9afe3b20a43d4b5bb800af39f945948d: '1c3fdc5272484ac09ef3e5e36a08f27c' -> '6dac90f01e744f9c8701a404440a4c77' move scheduled
I20240501 14:05:33.092089 7345 rebalancer_tool.cc:1074] tablet 9b88c43cb8704583a3e65c5c35c205f6: '04088ee3bea440a7bd5ff772152b9410' -> '938a24b0e6a040b7966225af26558ff8' move scheduled
I20240501 14:05:33.429652 7345 rebalancer_tool.cc:1074] tablet 23ca2bf1e59f44fcac8a48aff8d056b1: 'be56773105e744919601127a94e98adf' -> '19a6d2d8a08142d5bffcde3a37234692' move scheduled
I20240501 14:05:33.873168 7345 rebalancer_tool.cc:1074] tablet ba6cba753e934e719c730593c0706346: 'e9c60449cf3c46f8aca4ab6c535302a4' -> '4360b59e3a154688a79ca5f9c0a37bc9' move scheduled
I20240501 14:05:34.225405 7345 rebalancer_tool.cc:1074] tablet 61a274b24f53458f88c2cde4f8dbe4f4: '1c3fdc5272484ac09ef3e5e36a08f27c' -> '6dac90f01e744f9c8701a404440a4c77' move scheduled
I20240501 14:05:34.410106 7345 rebalancer_tool.cc:1074] tablet 51cbcf4888c440c2ad096c988ab2d968: '04088ee3bea440a7bd5ff772152b9410' -> '938a24b0e6a040b7966225af26558ff8' move scheduled
I20240501 14:05:34.962917 7345 rebalancer_tool.cc:1074] tablet 404e2851055541f0be1bb0971a7cc72a: 'be56773105e744919601127a94e98adf' -> '19a6d2d8a08142d5bffcde3a37234692' move scheduled
I20240501 14:05:35.269409 7345 rebalancer_tool.cc:1084] tablet f4d2b8c005b64bbb8e6621da068d15e5: 'e9c60449cf3c46f8aca4ab6c535302a4' -> '938a24b0e6a040b7966225af26558ff8' move ignored: Not found: Peer with uuid e9c60449cf3c46f8aca4ab6c535302a4 not found in consensus config
I20240501 14:05:35.269568 7345 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 14:05:37.854825 7345 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3c1513e944ee4b518373c4efa5e200cc: could not find any suitable replica to move from server e9c60449cf3c46f8aca4ab6c535302a4 to server 938a24b0e6a040b7966225af26558ff8
W20240501 14:05:37.855104 7345 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7af67fe405db4bffbc53cb288b8bb715: could not find any suitable replica to move from server be56773105e744919601127a94e98adf to server 6dac90f01e744f9c8701a404440a4c77
W20240501 14:05:37.855254 7345 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3794b1ccb5004f7cb92a229ef6ce3595: could not find any suitable replica to move from server e9c60449cf3c46f8aca4ab6c535302a4 to server 19a6d2d8a08142d5bffcde3a37234692
W20240501 14:05:37.855412 7345 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 403fdb578f8e4066a56f685925e83936: could not find any suitable replica to move from server be56773105e744919601127a94e98adf to server 6dac90f01e744f9c8701a404440a4c77
W20240501 14:05:37.855566 7345 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 763c884aecd640b49c65c692fbde9422: could not find any suitable replica to move from server 1c3fdc5272484ac09ef3e5e36a08f27c to server 6dac90f01e744f9c8701a404440a4c77
W20240501 14:05:37.868883 7345 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 41272b857ab045629ca60ab2467097b0: could not find any suitable replica to move from server e9c60449cf3c46f8aca4ab6c535302a4 to server 938a24b0e6a040b7966225af26558ff8
W20240501 14:05:37.872112 7345 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table facc024bfd644356a8879fd9661c928d: could not find any suitable replica to move from server be56773105e744919601127a94e98adf to server 19a6d2d8a08142d5bffcde3a37234692
W20240501 14:05:37.885008 7345 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5fc33bdfba464f65b68de5887d425ab7: could not find any suitable replica to move from server 1c3fdc5272484ac09ef3e5e36a08f27c to server 4360b59e3a154688a79ca5f9c0a37bc9
W20240501 14:05:37.886694 7345 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 84aeba643ddd44cfbbd6a34f81703482: could not find any suitable replica to move from server 04088ee3bea440a7bd5ff772152b9410 to server 938a24b0e6a040b7966225af26558ff8
W20240501 14:05:37.888130 7345 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 763c884aecd640b49c65c692fbde9422: could not find any suitable replica to move from server e9c60449cf3c46f8aca4ab6c535302a4 to server 6dac90f01e744f9c8701a404440a4c77
I20240501 14:05:38.193542 7345 rebalancer_tool.cc:1074] tablet c8a961e711114f369ba06840c062b982: '1c3fdc5272484ac09ef3e5e36a08f27c' -> '4360b59e3a154688a79ca5f9c0a37bc9' move scheduled
I20240501 14:05:38.390136 7345 rebalancer_tool.cc:1074] tablet 420642af520d45ffbc7f2d35f93e75a7: 'e9c60449cf3c46f8aca4ab6c535302a4' -> '938a24b0e6a040b7966225af26558ff8' move scheduled
I20240501 14:05:38.590904 7345 rebalancer_tool.cc:1074] tablet a726769d24284fedadd8e9e63cd27d05: '04088ee3bea440a7bd5ff772152b9410' -> '19a6d2d8a08142d5bffcde3a37234692' move scheduled
I20240501 14:05:38.741732 7345 rebalancer_tool.cc:1074] tablet ac4b234f794149e4b6def7a3b358942d: 'be56773105e744919601127a94e98adf' -> '6dac90f01e744f9c8701a404440a4c77' move scheduled
I20240501 14:05:38.990785 7345 rebalancer_tool.cc:1084] tablet 88ed5438fc934adc9fb98c6cb4476598: '1c3fdc5272484ac09ef3e5e36a08f27c' -> '4360b59e3a154688a79ca5f9c0a37bc9' move ignored: Illegal state: Request specified cas_config_opid_index of -1 but the committed config has opid_index of 4
I20240501 14:05:38.990916 7345 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 14:05:39.958840 9378 outbound_call.cc:322] RPC callback for RPC call kudu.master.MasterService.GetTabletLocations -> {remote=127.29.124.190:35761, user_credentials={real_user=slave}} blocked reactor thread for 61140.5us
W20240501 14:05:41.825399 7345 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 403fdb578f8e4066a56f685925e83936: could not find any suitable replica to move from server 4360b59e3a154688a79ca5f9c0a37bc9 to server 6dac90f01e744f9c8701a404440a4c77
W20240501 14:05:41.825516 7345 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3c1513e944ee4b518373c4efa5e200cc: could not find any suitable replica to move from server 6dac90f01e744f9c8701a404440a4c77 to server be56773105e744919601127a94e98adf
W20240501 14:05:41.836272 7345 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5fc33bdfba464f65b68de5887d425ab7: could not find any suitable replica to move from server be56773105e744919601127a94e98adf to server e9c60449cf3c46f8aca4ab6c535302a4
W20240501 14:05:41.836375 7345 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f26ba2f4e5654a5fa72021df5929200d: could not find any suitable replica to move from server 1c3fdc5272484ac09ef3e5e36a08f27c to server 4360b59e3a154688a79ca5f9c0a37bc9
W20240501 14:05:41.836457 7345 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7af67fe405db4bffbc53cb288b8bb715: could not find any suitable replica to move from server e9c60449cf3c46f8aca4ab6c535302a4 to server 938a24b0e6a040b7966225af26558ff8
W20240501 14:05:41.836566 7345 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 763c884aecd640b49c65c692fbde9422: could not find any suitable replica to move from server 1c3fdc5272484ac09ef3e5e36a08f27c to server 4360b59e3a154688a79ca5f9c0a37bc9
W20240501 14:05:41.836663 7345 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 41272b857ab045629ca60ab2467097b0: could not find any suitable replica to move from server e9c60449cf3c46f8aca4ab6c535302a4 to server 6dac90f01e744f9c8701a404440a4c77
W20240501 14:05:41.836716 7345 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3794b1ccb5004f7cb92a229ef6ce3595: could not find any suitable replica to move from server be56773105e744919601127a94e98adf to server 19a6d2d8a08142d5bffcde3a37234692
I20240501 14:05:42.135138 7345 rebalancer_tool.cc:1136] tablet ac4b234f794149e4b6def7a3b358942d: 'be56773105e744919601127a94e98adf' -> '6dac90f01e744f9c8701a404440a4c77' move completed: OK
I20240501 14:05:42.560366 7345 rebalancer_tool.cc:1136] tablet 1a7bbfc15a76450bbc9197037e856e08: 'e9c60449cf3c46f8aca4ab6c535302a4' -> '19a6d2d8a08142d5bffcde3a37234692' move completed: OK
I20240501 14:05:42.736567 7345 rebalancer_tool.cc:1136] tablet 404e2851055541f0be1bb0971a7cc72a: 'be56773105e744919601127a94e98adf' -> '19a6d2d8a08142d5bffcde3a37234692' move completed: OK
I20240501 14:05:42.809614 7345 rebalancer_tool.cc:1136] tablet 8cc5eec4600640f7a293877c825f8d96: 'be56773105e744919601127a94e98adf' -> '19a6d2d8a08142d5bffcde3a37234692' move completed: OK
I20240501 14:05:42.851845 7345 rebalancer_tool.cc:1136] tablet 420642af520d45ffbc7f2d35f93e75a7: 'e9c60449cf3c46f8aca4ab6c535302a4' -> '938a24b0e6a040b7966225af26558ff8' move completed: OK
I20240501 14:05:42.919245 7345 rebalancer_tool.cc:1136] tablet de6c7753c8564107b858876fcfa5924d: '04088ee3bea440a7bd5ff772152b9410' -> '6dac90f01e744f9c8701a404440a4c77' move completed: OK
I20240501 14:05:43.178972 7345 rebalancer_tool.cc:1136] tablet 9b88c43cb8704583a3e65c5c35c205f6: '04088ee3bea440a7bd5ff772152b9410' -> '938a24b0e6a040b7966225af26558ff8' move completed: OK
I20240501 14:05:43.237397 7345 rebalancer_tool.cc:1136] tablet fee16905387d428aa02cd2070c86c790: 'be56773105e744919601127a94e98adf' -> '4360b59e3a154688a79ca5f9c0a37bc9' move completed: OK
I20240501 14:05:43.359753 7345 rebalancer_tool.cc:1136] tablet 9d32e0707d244f2c9bbcb57f62b09f31: '04088ee3bea440a7bd5ff772152b9410' -> '6dac90f01e744f9c8701a404440a4c77' move completed: OK
I20240501 14:05:43.451028 7345 rebalancer_tool.cc:1136] tablet 51cbcf4888c440c2ad096c988ab2d968: '04088ee3bea440a7bd5ff772152b9410' -> '938a24b0e6a040b7966225af26558ff8' move completed: OK
I20240501 14:05:43.529633 7345 rebalancer_tool.cc:1136] tablet 2c39b53867e047faaf24cc6f87a8100c: '1c3fdc5272484ac09ef3e5e36a08f27c' -> '938a24b0e6a040b7966225af26558ff8' move completed: OK
I20240501 14:05:43.622220 7345 rebalancer_tool.cc:1136] tablet 9fd94635797d447cbdc9181ee91f465b: 'e9c60449cf3c46f8aca4ab6c535302a4' -> '4360b59e3a154688a79ca5f9c0a37bc9' move completed: OK
I20240501 14:05:43.720332 7345 rebalancer_tool.cc:1136] tablet 9afe3b20a43d4b5bb800af39f945948d: '1c3fdc5272484ac09ef3e5e36a08f27c' -> '6dac90f01e744f9c8701a404440a4c77' move completed: OK
I20240501 14:05:43.735433 7345 rebalancer_tool.cc:1136] tablet 23ca2bf1e59f44fcac8a48aff8d056b1: 'be56773105e744919601127a94e98adf' -> '19a6d2d8a08142d5bffcde3a37234692' move completed: OK
I20240501 14:05:43.854175 7345 rebalancer_tool.cc:1136] tablet ba6cba753e934e719c730593c0706346: 'e9c60449cf3c46f8aca4ab6c535302a4' -> '4360b59e3a154688a79ca5f9c0a37bc9' move completed: OK
I20240501 14:05:43.920941 7345 rebalancer_tool.cc:1136] tablet 61a274b24f53458f88c2cde4f8dbe4f4: '1c3fdc5272484ac09ef3e5e36a08f27c' -> '6dac90f01e744f9c8701a404440a4c77' move completed: OK
I20240501 14:05:44.032979 7345 rebalancer_tool.cc:1136] tablet c8a961e711114f369ba06840c062b982: '1c3fdc5272484ac09ef3e5e36a08f27c' -> '4360b59e3a154688a79ca5f9c0a37bc9' move completed: Incomplete: tablet c8a961e711114f369ba06840c062b982, TS 1c3fdc5272484ac09ef3e5e36a08f27c -> TS 4360b59e3a154688a79ca5f9c0a37bc9 move failed, target replica disappeared
I20240501 14:05:44.095453 7345 rebalancer_tool.cc:1084] tablet e7d5f02ccba34254a4d20951878b1214: 'be56773105e744919601127a94e98adf' -> '938a24b0e6a040b7966225af26558ff8' move ignored: Illegal state: tablet e7d5f02ccba34254a4d20951878b1214: replica 938a24b0e6a040b7966225af26558ff8 is already present
I20240501 14:05:44.097395 7345 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): unacceptable health status UNAVAILABLE
W20240501 14:05:45.067881 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:05:45.110055 7340 rebalancer_tool-test.cc:1260] rebalancer run info: Runtime error: /tmp/dist-test-taskUv7aek/build/asan/bin/kudu: process exited with non-zero status 1
stdout: Per-server replica distribution summary:
Statistic | Value
-----------------------+-----------
Minimum Replica Count | 0
Maximum Replica Count | 23
Average Replica Count | 11.250000
Per-table replica distribution summary:
Replica Skew | Value
--------------+----------
Minimum | 3
Maximum | 3
Average | 3.000000
stderr: W20240501 14:05:26.883682 7341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240501 14:05:26.884106 7341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240501 14:05:26.897490 7341 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240501 14:05:29.406735 7341 rebalancer_tool.cc:215] running whole-cluster rebalancing
W20240501 14:05:31.009056 7341 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table facc024bfd644356a8879fd9661c928d: could not find any suitable replica to move from server 04088ee3bea440a7bd5ff772152b9410 to server 19a6d2d8a08142d5bffcde3a37234692
W20240501 14:05:31.009653 7341 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3794b1ccb5004f7cb92a229ef6ce3595: could not find any suitable replica to move from server 1c3fdc5272484ac09ef3e5e36a08f27c to server 938a24b0e6a040b7966225af26558ff8
W20240501 14:05:31.010030 7341 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 763c884aecd640b49c65c692fbde9422: could not find any suitable replica to move from server be56773105e744919601127a94e98adf to server 6dac90f01e744f9c8701a404440a4c77
W20240501 14:05:31.010197 7341 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3c1513e944ee4b518373c4efa5e200cc: could not find any suitable replica to move from server e9c60449cf3c46f8aca4ab6c535302a4 to server 4360b59e3a154688a79ca5f9c0a37bc9
W20240501 14:05:31.010294 7341 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table facc024bfd644356a8879fd9661c928d: could not find any suitable replica to move from server 04088ee3bea440a7bd5ff772152b9410 to server 4360b59e3a154688a79ca5f9c0a37bc9
W20240501 14:05:31.010478 7341 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 403fdb578f8e4066a56f685925e83936: could not find any suitable replica to move from server be56773105e744919601127a94e98adf to server 6dac90f01e744f9c8701a404440a4c77
W20240501 14:05:31.010586 7341 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5fc33bdfba464f65b68de5887d425ab7: could not find any suitable replica to move from server 1c3fdc5272484ac09ef3e5e36a08f27c to server 938a24b0e6a040b7966225af26558ff8
W20240501 14:05:31.010697 7341 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 84aeba643ddd44cfbbd6a34f81703482: could not find any suitable replica to move from server 1c3fdc5272484ac09ef3e5e36a08f27c to server 6dac90f01e744f9c8701a404440a4c77
W20240501 14:05:31.010795 7341 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f26ba2f4e5654a5fa72021df5929200d: could not find any suitable replica to move from server e9c60449cf3c46f8aca4ab6c535302a4 to server 19a6d2d8a08142d5bffcde3a37234692
W20240501 14:05:31.010965 7341 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7af67fe405db4bffbc53cb288b8bb715: could not find any suitable replica to move from server 04088ee3bea440a7bd5ff772152b9410 to server 19a6d2d8a08142d5bffcde3a37234692
W20240501 14:05:31.011073 7341 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 763c884aecd640b49c65c692fbde9422: could not find any suitable replica to move from server 04088ee3bea440a7bd5ff772152b9410 to server 4360b59e3a154688a79ca5f9c0a37bc9
W20240501 14:05:31.011214 7341 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 41272b857ab045629ca60ab2467097b0: could not find any suitable replica to move from server 1c3fdc5272484ac09ef3e5e36a08f27c to server 938a24b0e6a040b7966225af26558ff8
W20240501 14:05:31.011368 7341 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 403fdb578f8e4066a56f685925e83936: could not find any suitable replica to move from server e9c60449cf3c46f8aca4ab6c535302a4 to server 6dac90f01e744f9c8701a404440a4c77
W20240501 14:05:31.011482 7341 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3c1513e944ee4b518373c4efa5e200cc: could not find any suitable replica to move from server 04088ee3bea440a7bd5ff772152b9410 to server 19a6d2d8a08142d5bffcde3a37234692
W20240501 14:05:31.011574 7341 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3794b1ccb5004f7cb92a229ef6ce3595: could not find any suitable replica to move from server be56773105e744919601127a94e98adf to server 4360b59e3a154688a79ca5f9c0a37bc9
I20240501 14:05:31.268155 7341 rebalancer_tool.cc:1084] tablet de6c7753c8564107b858876fcfa5924d: 'be56773105e744919601127a94e98adf' -> '6dac90f01e744f9c8701a404440a4c77' move ignored: Illegal state: tablet de6c7753c8564107b858876fcfa5924d: replica 6dac90f01e744f9c8701a404440a4c77 is already present
I20240501 14:05:31.268783 7341 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 14:05:33.673070 7341 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3794b1ccb5004f7cb92a229ef6ce3595: could not find any suitable replica to move from server 1c3fdc5272484ac09ef3e5e36a08f27c to server 19a6d2d8a08142d5bffcde3a37234692
W20240501 14:05:33.673380 7341 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table facc024bfd644356a8879fd9661c928d: could not find any suitable replica to move from server 04088ee3bea440a7bd5ff772152b9410 to server 19a6d2d8a08142d5bffcde3a37234692
W20240501 14:05:33.673508 7341 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3c1513e944ee4b518373c4efa5e200cc: could not find any suitable replica to move from server 04088ee3bea440a7bd5ff772152b9410 to server 19a6d2d8a08142d5bffcde3a37234692
W20240501 14:05:33.673621 7341 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 7af67fe405db4bffbc53cb288b8bb715: could not find any suitable replica to move from server be56773105e744919601127a94e98adf to server 4360b59e3a154688a79ca5f9c0a37bc9
W20240501 14:05:33.673755 7341 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f26ba2f4e5654a5fa72021df5929200d: could not find any suitable replica to move from server e9c60449cf3c46f8aca4ab6c535302a4 to server 4360b59e3a154688a79ca5f9c0a37bc9
W20240501 14:05:33.673934 7341 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5fc33bdfba464f65b68de5887d425ab7: could not find any suitable replica to move from server be56773105e744919601127a94e98adf to server 938a24b0e6a040b7966225af26558ff8
W20240501 14:05:33.674144 7341 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3794b1ccb5004f7cb92a229ef6ce3595: could not find any suitable replica to move from server e9c60449cf3c46f8aca4ab6c535302a4 to server 4360b59e3a154688a79ca5f9c0a37bc9
W20240501 14:05:33.674490 7341 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 403fdb578f8e4066a56f685925e83936: could not find any suitable replica to move from server be56773105e744919601127a94e98adf to server 4360b59e3a154688a79ca5f9c0a37bc9
W20240501 14:05:33.674846 7341 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 41272b857ab045629ca60ab2467097b0: could not find any suitable replica to move from server 1c3fdc5272484ac09ef3e5e36a08f27c to server 6dac90f01e744f9c8701a404440a4c77
W20240501 14:05:33.675303 7341 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 763c884aecd640b49c65c692fbde9422: could not find any suitable replica to move from server 04088ee3bea440a7bd5ff772152b9410 to server 19a6d2d8a08142d5bffcde3a37234692
W20240501 14:05:33.675845 7341 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 5fc33bdfba464f65b68de5887d425ab7: could not find any suitable replica to move from server 1c3fdc5272484ac09ef3e5e36a08f27c to server 6dac90f01e744f9c8701a404440a4c77
W20240501 14:05:33.676396 7341 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 41272b857ab045629ca60ab2467097b0: could not find any suitable replica to move from server be56773105e744919601127a94e98adf to server 19a6d2d8a08142d5bffcde3a37234692
W20240501 14:05:33.676828 7341 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 84aeba643ddd44cfbbd6a34f81703482: could not find any suitable replica to move from server e9c60449cf3c46f8aca4ab6c535302a4 to server 4360b59e3a154688a79ca5f9c0a37bc9
W20240501 14:05:33.677335 7341 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table facc024bfd644356a8879fd9661c928d: could not find any suitable replica to move from server 04088ee3bea440a7bd5ff772152b9410 to server 6dac90f01e744f9c8701a404440a4c77
W20240501 14:05:33.677742 7341 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3c1513e944ee4b518373c4efa5e200cc: could not find any suitable replica to move from server 04088ee3bea440a7bd5ff772152b9410 to server 938a24b0e6a040b7966225af26558ff8
W20240501 14:05:33.706199 7341 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 3794b1ccb5004f7cb92a229ef6ce3595: could not find any suitable replica to move from server 1c3fdc5272484ac09ef3e5e36a08f27c to server 4360b59e3a154688a79ca5f9c0a37bc9
W20240501 14:05:33.707823 7341 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 403fdb578f8e4066a56f685925e83936: could not find any suitable replica to move from server e9c60449cf3c46f8aca4ab6c535302a4 to server 19a6d2d8a08142d5bffcde3a37234692
W20240501 14:05:33.707990 7341 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 84aeba643ddd44cfbbd6a34f81703482: could not find any suitable replica to move from server be56773105e744919601127a94e98adf to server 4360b59e3a154688a79ca5f9c0a37bc9
W20240501 14:05:33.708101 7341 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table f26ba2f4e5654a5fa72021df5929200d: could not find any suitable replica to move from server 04088ee3bea440a7bd5ff772152b9410 to server 19a6d2d8a08142d5bffcde3a37234692
W20240501 14:05:33.708796 7341 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 763c884aecd640b49c65c692fbde9422: could not find any suitable replica to move from server 1c3fdc5272484ac09ef3e5e36a08f27c to server 938a24b0e6a040b7966225af26558ff8
I20240501 14:05:34.131381 7341 rebalancer_tool.cc:1074] tablet 61a274b24f53458f88c2cde4f8dbe4f4: 'e9c60449cf3c46f8aca4ab6c535302a4' -> '19a6d2d8a08142d5bffcde3a37234692' move scheduled
I20240501 14:05:34.375236 7341 rebalancer_tool.cc:1074] tablet 51cbcf4888c440c2ad096c988ab2d968: '1c3fdc5272484ac09ef3e5e36a08f27c' -> '4360b59e3a154688a79ca5f9c0a37bc9' move scheduled
I20240501 14:05:34.508486 7341 rebalancer_tool.cc:1074] tablet 71660575d94748d8a9329f909a42a0cd: 'be56773105e744919601127a94e98adf' -> '6dac90f01e744f9c8701a404440a4c77' move scheduled
I20240501 14:05:34.714212 7341 rebalancer_tool.cc:1074] tablet e7d5f02ccba34254a4d20951878b1214: '04088ee3bea440a7bd5ff772152b9410' -> '938a24b0e6a040b7966225af26558ff8' move scheduled
I20240501 14:05:35.242187 7341 rebalancer_tool.cc:1074] tablet 2c39b53867e047faaf24cc6f87a8100c: 'e9c60449cf3c46f8aca4ab6c535302a4' -> '6dac90f01e744f9c8701a404440a4c77' move scheduled
I20240501 14:05:35.487015 7341 rebalancer_tool.cc:1074] tablet 43b46fcf827c4275a47291afb107c290: '04088ee3bea440a7bd5ff772152b9410' -> '19a6d2d8a08142d5bffcde3a37234692' move scheduled
I20240501 14:05:35.722308 7341 rebalancer_tool.cc:1074] tablet 71bdb382b08341218c8bf2459fe65fc0: '1c3fdc5272484ac09ef3e5e36a08f27c' -> '4360b59e3a154688a79ca5f9c0a37bc9' move scheduled
I20240501 14:05:36.392433 7341 rebalancer_tool.cc:1074] tablet de6c7753c8564107b858876fcfa5924d: 'be56773105e744919601127a94e98adf' -> '938a24b0e6a040b7966225af26558ff8' move scheduled
I20240501 14:05:36.686028 7341 rebalancer_tool.cc:1084] tablet 1a7bbfc15a76450bbc9197037e856e08: 'e9c60449cf3c46f8aca4ab6c535302a4' -> '6dac90f01e744f9c8701a404440a4c77' move ignored: Not found: Peer with uuid e9c60449cf3c46f8aca4ab6c535302a4 not found in consensus config
I20240501 14:05:36.686139 7341 rebalancer_tool.cc:756] re-synchronizing cluster state
W20240501 14:05:38.372787 7341 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table facc024bfd644356a8879fd9661c928d: could not find any suitable replica to move from server 04088ee3bea440a7bd5ff772152b9410 to server 19a6d2d8a08142d5bffcde3a37234692
W20240501 14:05:38.373267 7341 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 763c884aecd640b49c65c692fbde9422: could not find any suitable replica to move from server 1c3fdc5272484ac09ef3e5e36a08f27c to server 6dac90f01e744f9c8701a404440a4c77
W20240501 14:05:38.373378 7341 rebalancer_tool.cc:1208] No replica could be moved this iteration: Not found: table 403fdb578f8e4066a56f685925e83936: could not find any suitable replica to move from server 1c3fdc5272484ac09ef3e5e36a08f27c to server 6dac90f01e744f9c8701a404440a4c77
I20240501 14:05:38.478667 7341 rebalancer_tool.cc:1074] tablet f4d2b8c005b64bbb8e6621da068d15e5: '1c3fdc5272484ac09ef3e5e36a08f27c' -> '4360b59e3a154688a79ca5f9c0a37bc9' move scheduled
I20240501 14:05:38.862958 7341 rebalancer_tool.cc:1074] tablet f9cb784f60774ea7b6e70ccfebf867a8: 'be56773105e744919601127a94e98adf' -> '19a6d2d8a08142d5bffcde3a37234692' move scheduled
I20240501 14:05:39.063285 7341 rebalancer_tool.cc:1074] tablet 63650547ab444f45bb377bcb49f258ad: 'e9c60449cf3c46f8aca4ab6c535302a4' -> '938a24b0e6a040b7966225af26558ff8' move scheduled
I20240501 14:05:39.545399 7341 rebalancer_tool.cc:1074] tablet 9b88c43cb8704583a3e65c5c35c205f6: '1c3fdc5272484ac09ef3e5e36a08f27c' -> '6dac90f01e744f9c8701a404440a4c77' move scheduled
I20240501 14:05:39.854157 7341 rebalancer_tool.cc:1074] tablet a726769d24284fedadd8e9e63cd27d05: '04088ee3bea440a7bd5ff772152b9410' -> '1c3fdc5272484ac09ef3e5e36a08f27c' move scheduled
I20240501 14:05:40.065191 7341 rebalancer_tool.cc:1074] tablet b284aa82594a44f5885f32fa05844913: 'be56773105e744919601127a94e98adf' -> '4360b59e3a154688a79ca5f9c0a37bc9' move scheduled
I20240501 14:05:40.755147 7341 rebalancer_tool.cc:1074] tablet 8cc5eec4600640f7a293877c825f8d96: 'e9c60449cf3c46f8aca4ab6c535302a4' -> '938a24b0e6a040b7966225af26558ff8' move scheduled
I20240501 14:05:41.066742 7341 rebalancer_tool.cc:1074] tablet 420642af520d45ffbc7f2d35f93e75a7: 'be56773105e744919601127a94e98adf' -> '6dac90f01e744f9c8701a404440a4c77' move scheduled
I20240501 14:05:41.761772 7341 rebalancer_tool.cc:1074] tablet 9d32e0707d244f2c9bbcb57f62b09f31: 'e9c60449cf3c46f8aca4ab6c535302a4' -> '4360b59e3a154688a79ca5f9c0a37bc9' move scheduled
I20240501 14:05:42.120404 7341 rebalancer_tool.cc:1136] tablet 43b46fcf827c4275a47291afb107c290: '04088ee3bea440a7bd5ff772152b9410' -> '19a6d2d8a08142d5bffcde3a37234692' move completed: OK
I20240501 14:05:42.426270 7341 rebalancer_tool.cc:1136] tablet 2c39b53867e047faaf24cc6f87a8100c: 'e9c60449cf3c46f8aca4ab6c535302a4' -> '6dac90f01e744f9c8701a404440a4c77' move completed: OK
I20240501 14:05:42.590890 7341 rebalancer_tool.cc:1136] tablet 71bdb382b08341218c8bf2459fe65fc0: '1c3fdc5272484ac09ef3e5e36a08f27c' -> '4360b59e3a154688a79ca5f9c0a37bc9' move completed: OK
I20240501 14:05:43.003996 7341 rebalancer_tool.cc:1136] tablet 71660575d94748d8a9329f909a42a0cd: 'be56773105e744919601127a94e98adf' -> '6dac90f01e744f9c8701a404440a4c77' move completed: Incomplete: tablet 71660575d94748d8a9329f909a42a0cd, TS be56773105e744919601127a94e98adf -> TS 6dac90f01e744f9c8701a404440a4c77 move failed, target replica disappeared
I20240501 14:05:43.042706 7341 rebalancer_tool.cc:1136] tablet 61a274b24f53458f88c2cde4f8dbe4f4: 'e9c60449cf3c46f8aca4ab6c535302a4' -> '19a6d2d8a08142d5bffcde3a37234692' move completed: OK
I20240501 14:05:43.437171 7341 rebalancer_tool.cc:1136] tablet a726769d24284fedadd8e9e63cd27d05: '04088ee3bea440a7bd5ff772152b9410' -> '1c3fdc5272484ac09ef3e5e36a08f27c' move completed: Incomplete: tablet a726769d24284fedadd8e9e63cd27d05, TS 04088ee3bea440a7bd5ff772152b9410 -> TS 1c3fdc5272484ac09ef3e5e36a08f27c move failed, target replica disappeared
I20240501 14:05:43.492013 7341 rebalancer_tool.cc:1136] tablet b284aa82594a44f5885f32fa05844913: 'be56773105e744919601127a94e98adf' -> '4360b59e3a154688a79ca5f9c0a37bc9' move completed: Incomplete: tablet b284aa82594a44f5885f32fa05844913, TS be56773105e744919601127a94e98adf -> TS 4360b59e3a154688a79ca5f9c0a37bc9 move failed, target replica disappeared
I20240501 14:05:43.570732 7341 rebalancer_tool.cc:1136] tablet f4d2b8c005b64bbb8e6621da068d15e5: '1c3fdc5272484ac09ef3e5e36a08f27c' -> '4360b59e3a154688a79ca5f9c0a37bc9' move completed: OK
I20240501 14:05:43.711624 7341 rebalancer_tool.cc:1136] tablet 63650547ab444f45bb377bcb49f258ad: 'e9c60449cf3c46f8aca4ab6c535302a4' -> '938a24b0e6a040b7966225af26558ff8' move completed: OK
I20240501 14:05:43.730579 7341 rebalancer_tool.cc:1136] tablet de6c7753c8564107b858876fcfa5924d: 'be56773105e744919601127a94e98adf' -> '938a24b0e6a040b7966225af26558ff8' move completed: OK
I20240501 14:05:43.947237 7341 rebalancer_tool.cc:1136] tablet 9b88c43cb8704583a3e65c5c35c205f6: '1c3fdc5272484ac09ef3e5e36a08f27c' -> '6dac90f01e744f9c8701a404440a4c77' move completed: OK
I20240501 14:05:44.037837 7341 rebalancer_tool.cc:1136] tablet 8cc5eec4600640f7a293877c825f8d96: 'e9c60449cf3c46f8aca4ab6c535302a4' -> '938a24b0e6a040b7966225af26558ff8' move completed: OK
I20240501 14:05:44.093662 7341 rebalancer_tool.cc:1136] tablet f9cb784f60774ea7b6e70ccfebf867a8: 'be56773105e744919601127a94e98adf' -> '19a6d2d8a08142d5bffcde3a37234692' move completed: OK
I20240501 14:05:44.183938 7341 rebalancer_tool.cc:1084] tablet 3d6859d53fd747a5a5e94722626fa522: '1c3fdc5272484ac09ef3e5e36a08f27c' -> '19a6d2d8a08142d5bffcde3a37234692' move ignored: Illegal state: tablet 3d6859d53fd747a5a5e94722626fa522: replica 19a6d2d8a08142d5bffcde3a37234692 is already present
I20240501 14:05:44.184736 7341 rebalancer_tool.cc:756] re-synchronizing cluster state
Illegal state: tablet server 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): unacceptable health status UNAVAILABLE
W20240501 14:05:45.139976 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:05:45.231716 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:05:45.274070 4840 catalog_manager.cc:4573] TS 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 1a7bbfc15a76450bbc9197037e856e08: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:05:45.281539 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:05:45.362107 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:05:45.390002 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:05:45.391860 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:05:45.403764 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:05:45.457691 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:05:45.694577 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:05:45.759799 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
I20240501 14:05:45.819243 9764 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:05:45.819521 9764 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:05:45.820405 9764 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers be56773105e744919601127a94e98adf (127.29.124.132:35459), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:05:45.820837 5386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf" is_pre_election: true
I20240501 14:05:45.821102 5386 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 2.
W20240501 14:05:45.822196 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:05:45.822539 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:05:45.822664 5191 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:05:45.823156 9764 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:05:45.824306 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:05:45.851522 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:05:45.889153 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:05:45.894860 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:05:45.938795 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:05:46.029523 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:05:46.054656 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:05:46.058075 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:05:46.058094 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:05:46.105423 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:05:46.110450 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:05:46.268862 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:05:46.445124 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:05:46.459136 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:05:46.746785 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:05:46.753813 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:05:46.849712 4840 catalog_manager.cc:4573] TS 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 1a7bbfc15a76450bbc9197037e856e08: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:05:46.932835 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:05:47.187769 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:05:47.221657 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:05:47.428912 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:05:47.439718 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:05:47.509711 10443 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:05:47.509910 10443 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:05:47.510903 10443 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers be56773105e744919601127a94e98adf (127.29.124.132:35459), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:05:47.511528 5386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf" is_pre_election: true
I20240501 14:05:47.511754 5386 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 2.
W20240501 14:05:47.512885 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:05:47.513092 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:05:47.513177 5191 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:05:47.513453 10443 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:05:47.638453 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:05:47.683091 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:05:47.683786 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:05:47.781452 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:05:47.853928 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:05:47.898211 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:05:47.944300 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:05:48.040081 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20240501 14:05:48.093683 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 14:05:48.143532 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 14:05:48.144649 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:05:48.188926 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 14:05:48.200461 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:05:48.242086 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:05:48.261126 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:05:48.431790 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:05:48.465678 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:05:48.465864 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:05:48.490680 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:05:48.517354 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:05:48.601516 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:05:48.668011 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:05:48.693130 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:05:48.779016 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:05:48.791997 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:05:48.842865 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:05:48.900005 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:05:48.940168 4840 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 16 similar messages]
W20240501 14:05:48.941658 4840 catalog_manager.cc:4573] TS 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 1a7bbfc15a76450bbc9197037e856e08: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:05:48.947736 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:05:49.070168 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:05:49.294821 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:05:49.456219 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:05:49.735611 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:05:49.838371 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:05:49.928885 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:05:50.073550 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:05:50.156754 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:05:50.183943 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20240501 14:05:50.252676 10808 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:05:50.252856 10808 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:05:50.253851 10808 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:05:50.254309 5254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" is_pre_election: true
I20240501 14:05:50.254549 5254 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 2.
W20240501 14:05:50.255625 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:05:50.255825 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:05:50.255939 5321 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:05:50.256233 10808 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:05:50.307694 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:05:50.367245 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:05:50.451555 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:05:50.509303 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:05:50.528226 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:05:50.589277 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:05:50.731452 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:05:50.775352 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:05:50.855209 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:05:50.885704 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:05:50.941221 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:05:50.952960 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:05:50.992781 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:05:51.035669 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:05:51.126428 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:05:51.181344 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:05:51.294744 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:05:51.306918 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:05:51.340600 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:05:51.504530 4840 catalog_manager.cc:4573] TS 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 8cc5eec4600640f7a293877c825f8d96: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:05:51.542809 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:05:51.708456 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:05:51.723824 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:05:51.794410 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:05:51.954227 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:05:51.956701 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:05:52.274649 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:05:52.339202 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:05:52.458158 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:05:52.593107 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:05:52.657208 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:05:52.715202 11041 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:05:52.715369 11041 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:05:52.716169 11041 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:05:52.716467 5254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" is_pre_election: true
I20240501 14:05:52.716699 5254 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 2.
W20240501 14:05:52.717742 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:05:52.717967 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:05:52.718097 5321 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:05:52.718353 11041 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:05:52.763693 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:05:52.867648 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:05:52.896966 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:05:52.998720 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:05:53.000339 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:05:53.019310 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:05:53.062474 4840 catalog_manager.cc:4573] TS 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 1a7bbfc15a76450bbc9197037e856e08: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:05:53.063035 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:05:53.146703 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 14:05:53.194095 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240501 14:05:53.194926 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:05:53.249972 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 14:05:53.279042 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:05:53.279356 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 14:05:53.321023 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:05:53.393018 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:05:53.434545 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:05:53.516641 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:05:53.602313 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:05:53.666916 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:05:53.679209 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:05:53.785480 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:05:53.874722 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:05:53.996258 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:05:54.112947 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:05:54.129570 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:05:54.140311 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:05:54.348919 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:05:54.499752 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:05:54.575484 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:05:54.577666 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:05:54.789992 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:05:54.966404 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:05:55.025120 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:05:55.049861 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:05:55.141212 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:05:55.200915 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:05:55.486496 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:05:55.526059 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:05:55.538424 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:05:55.540827 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:05:55.657197 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:05:55.821110 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:05:55.832875 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:05:55.843559 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:05:55.845053 11299 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:05:55.845221 11299 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
W20240501 14:05:55.846067 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20240501 14:05:55.846145 11299 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:05:55.846438 5254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" is_pre_election: true
I20240501 14:05:55.846664 5254 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 2.
W20240501 14:05:55.847935 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:05:55.855899 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:05:55.855995 5321 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:05:55.856215 11299 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:05:55.907815 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:05:55.931329 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:05:55.971238 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:05:55.992522 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:05:56.036235 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:05:56.143457 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:05:56.149503 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:05:56.427875 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:05:56.454372 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:05:56.549799 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:05:56.565949 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:05:56.678948 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:05:56.723095 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:05:56.775315 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:05:56.988771 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:05:56.995715 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:05:57.356933 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:05:57.378504 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:05:57.400863 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:05:57.463238 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:05:57.463984 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:05:57.777473 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:05:57.851044 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:05:57.852061 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:05:57.954309 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:05:57.962468 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:05:57.984946 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:05:58.121937 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:05:58.126323 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:05:58.191473 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 74 similar messages]
W20240501 14:05:58.206380 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 14:05:58.207304 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:05:58.233688 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:05:58.387079 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 14:05:58.426194 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 14:05:58.427047 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:05:58.438544 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:05:58.450891 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:05:58.566845 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:05:58.607471 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:05:58.705369 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:05:58.797042 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:05:58.815474 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:05:58.922386 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:05:59.084570 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:05:59.121248 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:05:59.172885 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:05:59.206138 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:05:59.299819 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:05:59.375350 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:05:59.413461 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:05:59.585901 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:05:59.734616 4840 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 2 similar messages]
W20240501 14:05:59.735617 4840 catalog_manager.cc:4573] TS 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 8cc5eec4600640f7a293877c825f8d96: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:05:59.819855 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:05:59.854933 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
I20240501 14:05:59.953898 11590 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:05:59.954110 11590 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:05:59.954926 11590 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers be56773105e744919601127a94e98adf (127.29.124.132:35459), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:05:59.955296 5384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf" is_pre_election: true
I20240501 14:05:59.955513 5384 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 2.
W20240501 14:05:59.956852 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:05:59.957043 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:05:59.957134 5191 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:05:59.957456 11590 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:05:59.998821 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:05:59.999967 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:00.108827 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:06:00.296171 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:06:00.490782 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:06:00.530035 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:00.596818 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:00.655337 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:06:00.677175 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:06:00.679116 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:06:00.756167 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:06:00.831703 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:06:00.878875 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:06:00.914312 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:06:00.961774 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:06:01.105005 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:06:01.105574 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:01.130249 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:01.268855 4840 catalog_manager.cc:4573] TS 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 1a7bbfc15a76450bbc9197037e856e08: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:01.317023 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:06:01.337414 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:06:01.536492 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:06:01.548384 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:01.562891 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:06:01.623557 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:01.795046 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:06:01.858824 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:06:01.862080 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:06:02.076719 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:02.123099 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:02.293110 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:06:02.365653 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:06:02.635766 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:02.700363 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:02.748526 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:06:02.841274 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:06:02.857635 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:06:03.076303 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:03.100258 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:06:03.106828 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:06:03.183851 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:06:03.240044 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 14:06:03.282380 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:03.358366 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 37 similar messages]
W20240501 14:06:03.368829 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:06:03.424985 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240501 14:06:03.442946 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:06:03.514721 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240501 14:06:03.558147 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:06:03.587893 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:06:03.618392 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:03.704421 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:06:03.726614 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:06:03.745568 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:06:03.765003 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:03.911502 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:06:03.936875 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:06:03.991775 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
I20240501 14:06:04.132912 12027 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:06:04.133077 12027 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:06:04.133929 12027 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers be56773105e744919601127a94e98adf (127.29.124.132:35459), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:06:04.134326 5386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf" is_pre_election: true
I20240501 14:06:04.134568 5386 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 2.
W20240501 14:06:04.136761 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:04.136919 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:06:04.136986 5191 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:06:04.137212 12027 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:06:04.171046 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:04.213649 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:04.218142 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:06:04.284162 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:06:04.634172 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:06:04.642426 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:06:04.668243 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:04.720384 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:05.118638 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:06:05.246503 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:05.313887 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:05.326877 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:06:05.476801 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:06:05.499402 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:06:05.591382 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:06:05.611267 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:06:05.690306 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:05.758714 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:05.893568 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:06:05.904124 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:06:05.951103 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:06:06.034253 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:06:06.074543 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:06:06.130100 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:06:06.169416 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:06.174005 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:06:06.217685 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:06:06.372299 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:06.495918 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:06:06.518299 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:06:06.586656 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:06:06.682451 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:06.790971 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:06:06.801192 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:06:06.860291 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:06:06.977779 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:07.233458 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:07.262027 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:06:07.570452 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:07.591953 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
I20240501 14:06:07.621529 12278 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:06:07.621745 12278 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:06:07.622453 12278 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers be56773105e744919601127a94e98adf (127.29.124.132:35459), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:06:07.623018 5386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf" is_pre_election: true
I20240501 14:06:07.623246 5386 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 2.
W20240501 14:06:07.623957 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:07.624089 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:06:07.624142 5191 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:06:07.624313 12278 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:06:07.790436 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:07.967370 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:06:07.995026 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:06:08.022580 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:06:08.037068 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:06:08.078178 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:08.189778 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:08.280077 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 14:06:08.281616 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:06:08.341331 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:06:08.396966 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 14:06:08.397956 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:06:08.442586 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240501 14:06:08.453919 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:06:08.462982 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:06:08.483624 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:08.523093 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 14:06:08.553808 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:06:08.561398 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:06:08.676499 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:06:08.689513 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:08.817804 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:06:08.919288 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:06:08.974872 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:06:09.095007 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:09.115108 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:06:09.197916 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:06:09.238757 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:06:09.245179 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:06:09.312783 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:09.489338 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:09.713696 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:06:09.816396 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:06:09.861812 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:09.879734 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:10.253726 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:10.281431 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:10.306218 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:06:10.398284 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:06:10.401568 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:06:10.456867 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:06:10.578414 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:06:10.698968 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:10.815654 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:06:10.816653 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:10.830371 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:06:10.861665 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:06:10.932476 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:06:11.032130 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:06:11.054208 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:06:11.058917 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:06:11.082502 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:11.269770 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:11.274117 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:06:11.275142 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:06:11.380092 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
I20240501 14:06:11.410714 12773 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:06:11.410892 12773 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:06:11.411816 12773 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:06:11.412127 5254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" is_pre_election: true
I20240501 14:06:11.412312 5254 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 2.
W20240501 14:06:11.413348 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:11.413535 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:06:11.413625 5321 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:06:11.413823 12773 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:06:11.593441 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:11.659091 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:06:11.728317 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:06:11.729578 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:06:11.854252 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:06:11.881603 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:12.030998 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:12.070524 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:06:12.257140 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:06:12.324038 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:12.466847 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:12.795143 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:06:12.812219 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:06:12.863751 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:06:12.876293 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:12.979974 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:13.099393 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:06:13.240932 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:06:13.284034 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240501 14:06:13.356532 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:06:13.429829 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:06:13.430791 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:06:13.463495 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:13.466653 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:06:13.494894 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 14:06:13.541198 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:13.580622 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 14:06:13.581475 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:06:13.584141 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:06:13.596577 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:06:13.626716 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:06:13.742666 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:06:13.752367 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:06:13.834183 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:06:14.027099 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:14.064954 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:06:14.119860 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:14.342908 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:06:14.376029 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:06:14.473420 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:06:14.512087 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:14.575623 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:06:14.624418 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:14.724467 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:06:14.947207 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:15.092427 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:15.230548 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:06:15.303550 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:06:15.348858 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:15.421823 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:06:15.512192 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:15.617770 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:06:15.837846 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:06:15.861850 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:06:15.886072 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:06:15.926774 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:06:15.938617 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:15.974865 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:06:15.992583 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:06:16.088361 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:06:16.089331 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:16.112195 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:06:16.139390 4840 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240501 14:06:16.140540 4840 catalog_manager.cc:4573] TS 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 8cc5eec4600640f7a293877c825f8d96: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:16.242489 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:06:16.380468 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:16.430429 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:06:16.458355 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:06:16.602955 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:06:16.690408 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:16.817040 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:16.846407 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:06:16.885921 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:06:16.923070 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:06:17.065366 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:06:17.154424 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:06:17.217954 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:17.234351 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:17.663125 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:06:17.672598 4840 catalog_manager.cc:4573] TS 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 1a7bbfc15a76450bbc9197037e856e08: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:17.714217 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:06:17.766561 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:17.778481 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:18.206583 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:06:18.230746 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:06:18.280714 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:06:18.322228 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:18.346606 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:06:18.368023 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:18.377497 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:06:18.425246 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 14:06:18.426050 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:06:18.472376 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:06:18.559018 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 14:06:18.581864 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:06:18.616160 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240501 14:06:18.617549 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:06:18.660261 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:06:18.690419 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 14:06:18.700712 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:06:18.776858 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:18.845021 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:06:18.916247 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:18.958459 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:06:19.190506 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:19.223752 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:06:19.260743 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:06:19.269318 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:06:19.364370 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:06:19.534626 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:19.566809 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:06:19.601109 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:06:19.728531 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:19.999708 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:06:20.089001 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:20.142702 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:06:20.224964 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:20.536779 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:20.612052 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:06:20.660100 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:06:20.689488 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:06:20.727979 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:20.860690 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:06:20.914086 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:06:20.941082 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:06:20.955705 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:06:20.962803 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:06:20.982837 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:21.032714 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:06:21.174415 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:06:21.177862 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:06:21.189492 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
I20240501 14:06:21.234706 13319 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:06:21.234918 13319 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:06:21.235661 13319 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:06:21.236032 5254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" is_pre_election: true
I20240501 14:06:21.236241 5254 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 2.
W20240501 14:06:21.237324 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:21.237489 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:06:21.237579 5321 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:06:21.237808 13319 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:06:21.305447 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:06:21.426409 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:21.549209 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:06:21.592423 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:21.675494 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:06:21.774274 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:06:21.788396 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:06:21.910662 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:06:21.974377 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:22.029996 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:22.086812 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:06:22.136744 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:06:22.386153 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:22.417212 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:22.454478 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:06:22.672376 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:06:22.786026 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:22.813565 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:22.942430 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:06:23.136081 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:06:23.204114 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:23.218199 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:06:23.223923 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:06:23.235702 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:06:23.389246 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:06:23.431252 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:23.485082 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240501 14:06:23.560475 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:06:23.612269 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:06:23.614275 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:06:23.683228 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:06:23.702006 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 14:06:23.713567 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 14:06:23.738281 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 14:06:23.739194 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:06:23.757886 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:23.772856 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:06:23.992103 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:24.001902 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:06:24.245718 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:06:24.255864 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:24.387313 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:06:24.391042 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:24.400830 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:06:24.551155 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:06:24.649070 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:06:24.798193 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:06:24.847411 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:24.880692 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:06:24.934582 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:25.227447 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:25.434589 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:06:25.535867 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:25.591641 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:06:25.601950 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:06:25.635700 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:25.643767 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:06:25.675529 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:06:25.795620 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:06:25.902551 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:06:25.990744 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:06:26.063880 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:26.087306 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:06:26.165656 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:26.188030 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:06:26.192848 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:06:26.297967 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:06:26.317838 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:06:26.369900 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:06:26.583127 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:06:26.626966 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:26.699252 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:26.977169 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:06:27.007870 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:06:27.057529 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:06:27.165167 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:27.185706 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:06:27.240964 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:06:27.276749 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:06:27.284384 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:06:27.304735 14149 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:06:27.304926 14149 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:06:27.305858 14149 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:06:27.306277 5254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" is_pre_election: true
I20240501 14:06:27.306519 5254 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 2.
W20240501 14:06:27.307775 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:27.307926 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:06:27.308022 5321 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:06:27.308283 14149 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:06:27.753868 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:27.836519 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:06:27.842180 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:27.885293 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:06:28.041954 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:06:28.135967 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:28.180997 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:06:28.216785 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:06:28.259805 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:28.313828 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:06:28.323896 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:06:28.361743 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:06:28.377410 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:06:28.434468 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:06:28.595240 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 74 similar messages]
W20240501 14:06:28.649104 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:28.678855 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:06:28.705948 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240501 14:06:28.723459 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:28.763582 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 14:06:28.766817 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:06:28.806468 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:06:28.860395 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:06:28.861207 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:06:28.979825 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:06:29.137948 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:29.147303 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:29.357893 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:06:29.480480 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:06:29.508949 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:06:29.575273 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:06:29.628152 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:29.699412 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:06:29.704975 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:29.929941 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:06:30.094897 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:30.236207 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:06:30.240236 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:06:30.284268 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:30.559202 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:06:30.562916 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:06:30.637053 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:30.695256 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:06:30.711409 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:06:30.858836 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:06:30.886541 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:30.892021 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:06:30.892114 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:06:31.016189 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:06:31.039757 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:06:31.111217 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:06:31.121798 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:31.212774 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:06:31.246608 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:06:31.266062 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:31.368265 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:06:31.637907 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:31.726724 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:31.974064 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:06:32.019932 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:06:32.046700 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:06:32.137892 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:32.246863 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:32.256361 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:06:32.469077 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:06:32.613775 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:06:32.704370 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:32.830544 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:33.040553 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:06:33.116559 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:06:33.156623 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:06:33.157995 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:06:33.188868 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:06:33.200214 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:33.228442 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:06:33.290644 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:06:33.325222 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:06:33.354625 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:33.496701 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:06:33.529803 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:06:33.539855 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:06:33.647516 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:33.668774 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:06:33.670641 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 14:06:33.737429 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:06:33.756186 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 14:06:33.877066 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:33.879844 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:06:33.912750 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 14:06:33.923933 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:06:34.040180 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:06:34.055564 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:34.343930 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:06:34.393916 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:06:34.446882 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:06:34.476344 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:34.633715 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:34.819694 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:06:34.916208 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:35.023195 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:06:35.040895 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:06:35.105026 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:35.329794 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:06:35.489048 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:06:35.511073 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:35.643296 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:06:35.646077 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:06:35.666468 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:35.697676 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:06:35.755944 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:06:35.834372 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:06:35.835026 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:06:35.893476 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:06:35.999440 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:06:36.039907 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:06:36.110935 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:36.206059 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:06:36.245893 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:06:36.274307 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:36.421778 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:06:36.501430 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:06:36.655767 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:06:36.672698 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:36.729441 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:36.969564 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:06:37.057453 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:06:37.266188 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:37.316576 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:37.342638 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:06:37.424439 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:06:37.513288 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:06:37.791297 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:37.814116 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:37.841975 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:06:37.908759 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:06:38.064764 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:06:38.101133 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:06:38.169128 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:38.181638 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:06:38.210325 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:38.231984 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:06:38.244467 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:06:38.273586 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:06:38.373844 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:06:38.462627 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:06:38.600221 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:38.672528 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:06:38.711885 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:38.741210 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 14:06:38.771587 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:06:38.776739 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 14:06:38.884141 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:06:38.906045 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:06:38.941242 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240501 14:06:38.990514 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:06:39.077142 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:39.095220 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:06:39.100950 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:06:39.215160 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:39.320483 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:06:39.434710 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:06:39.526898 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:39.664456 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:39.758200 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:06:39.921763 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:39.932678 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240501 14:06:40.091225 15125 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:06:40.091418 15125 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:06:40.092514 15125 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers be56773105e744919601127a94e98adf (127.29.124.132:35459), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:06:40.093005 5386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf" is_pre_election: true
I20240501 14:06:40.093330 5386 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 2.
W20240501 14:06:40.095677 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:40.095949 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:06:40.096025 5191 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:06:40.096235 15125 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:06:40.099040 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:06:40.218345 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:40.378491 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:06:40.384526 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:06:40.395543 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:06:40.456082 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:40.736717 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:06:40.763073 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:06:40.765534 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:40.768066 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:06:40.776440 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:06:40.791224 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:06:40.815562 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:06:40.837292 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:06:41.043080 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:41.163540 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:41.165470 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:06:41.187067 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:06:41.298771 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:06:41.368546 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:06:41.477663 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:41.484087 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:06:41.564256 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:06:41.657076 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:41.723901 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:06:41.793767 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:06:41.861510 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:42.217526 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:42.249684 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:42.357470 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:06:42.417160 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:06:42.612426 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:42.650549 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:06:42.677445 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:06:42.787783 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:06:42.791311 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:42.827674 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:06:43.049597 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:43.184849 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:06:43.195350 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:06:43.210785 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:06:43.297566 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:06:43.298612 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:06:43.303841 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:06:43.349112 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:06:43.389971 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:43.520677 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:43.720378 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:06:43.769088 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 14:06:43.807802 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 63 similar messages]
W20240501 14:06:43.808686 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:06:43.812505 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:06:43.825958 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:06:43.867142 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:06:43.880263 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:44.008010 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:06:44.045536 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:44.139014 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:06:44.158839 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 14:06:44.237416 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:06:44.293989 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:06:44.399052 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:44.579613 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:44.840821 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:44.879947 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:06:44.961318 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:06:45.022907 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:45.062669 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:06:45.171589 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:06:45.287767 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:45.461804 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:06:45.465492 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:06:45.486290 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:06:45.539330 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:06:45.539605 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:06:45.565739 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:45.630883 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:06:45.732851 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:06:45.773859 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:06:45.787513 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:06:45.902668 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:45.952215 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:46.168594 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:06:46.293102 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:06:46.300879 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:06:46.335374 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:06:46.368949 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:06:46.386157 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:46.406632 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:46.551314 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:06:46.713968 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:06:46.943920 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:46.964938 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:47.054170 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:06:47.286645 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:06:47.405393 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:47.452294 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:06:47.511464 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:47.646869 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:06:47.681636 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:06:47.688254 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:06:47.759202 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:06:47.816951 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:06:47.919695 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:47.944911 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:47.989984 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:06:48.121204 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:06:48.187669 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:06:48.195731 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:06:48.310985 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:06:48.419124 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:06:48.487087 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:48.496390 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:48.647570 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:06:48.831876 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 14:06:48.832769 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:06:48.841331 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 57 similar messages]
W20240501 14:06:48.842279 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:06:48.939322 4840 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240501 14:06:48.940408 4840 catalog_manager.cc:4573] TS 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 8cc5eec4600640f7a293877c825f8d96: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
I20240501 14:06:48.952977 16085 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:06:48.953195 16085 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:06:48.954165 16085 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers be56773105e744919601127a94e98adf (127.29.124.132:35459), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:06:48.954542 5386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf" is_pre_election: true
I20240501 14:06:48.954774 5386 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 2.
W20240501 14:06:48.955371 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:48.955785 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:48.955961 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:06:48.956066 5191 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:06:48.956310 16085 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:06:48.977334 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:06:49.042937 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:49.048125 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:06:49.207978 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:06:49.251230 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 14:06:49.387133 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 14:06:49.393573 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:06:49.524288 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:06:49.532001 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:49.619627 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:49.743376 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:06:49.777161 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:06:49.916121 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:06:50.022909 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:50.159058 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:06:50.159689 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:50.237315 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:06:50.406957 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:06:50.475756 4840 catalog_manager.cc:4573] TS 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 1a7bbfc15a76450bbc9197037e856e08: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:50.537364 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:06:50.557595 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:50.585909 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:06:50.587600 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:06:50.651895 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:06:50.709326 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:06:50.780144 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:50.900732 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:06:50.917845 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:06:50.996325 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:06:51.023321 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:51.102632 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:06:51.197571 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:06:51.256525 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:51.521550 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:51.538482 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:06:51.540120 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:06:51.620030 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:06:51.641855 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:51.751482 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:06:52.019342 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:52.040699 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:52.124282 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:06:52.180141 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:06:52.215821 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:06:52.370891 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:06:52.452095 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:52.460646 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:52.688513 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:06:52.814594 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:06:52.866601 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:06:52.870671 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:52.944651 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:06:52.986882 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:06:53.018177 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:06:53.058710 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:06:53.075484 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:53.192056 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:06:53.245836 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:06:53.325368 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:06:53.440637 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:53.495316 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:53.584838 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:06:53.592324 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:06:53.645658 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:06:53.902179 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 14:06:53.903028 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:06:53.915253 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240501 14:06:53.958256 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:06:53.984622 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:54.039870 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:54.084633 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:06:54.134969 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:06:54.408594 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 63 similar messages]
W20240501 14:06:54.409484 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:06:54.453878 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 14:06:54.459709 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:54.539783 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:54.690580 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:06:54.854387 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:06:54.871521 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:06:54.939911 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:54.951311 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:54.970542 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:06:55.237448 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:06:55.252174 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:06:55.362453 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:55.379926 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:06:55.446724 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:06:55.453817 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:55.465884 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:06:55.567740 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:06:55.629714 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:06:55.823436 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:06:55.929067 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:55.949204 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:56.004722 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:06:56.089174 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:06:56.192245 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:06:56.257390 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:06:56.291097 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:06:56.386556 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:56.476210 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:06:56.541312 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:56.578841 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:06:56.767629 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:06:56.921768 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:56.975706 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:06:57.098404 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:06:57.165380 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
I20240501 14:06:57.303664 16898 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:06:57.303831 16898 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:06:57.304723 16898 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers be56773105e744919601127a94e98adf (127.29.124.132:35459), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:06:57.305008 5386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf" is_pre_election: true
I20240501 14:06:57.305207 5386 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 2.
W20240501 14:06:57.307044 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:57.307384 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:06:57.307487 5191 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:06:57.307693 16898 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:06:57.309216 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:06:57.347762 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:57.581513 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:06:57.614391 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:06:57.661751 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:06:57.687551 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:06:57.691202 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:06:57.769449 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:57.805604 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:57.858657 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:06:57.991225 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:06:58.019551 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:06:58.063275 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:06:58.146847 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:58.333817 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:58.343987 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:06:58.360872 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:06:58.509490 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:06:58.613678 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:58.777530 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:06:58.781888 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:58.804221 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:06:58.847867 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:06:58.887408 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:06:58.912539 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 14:06:58.930687 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 63 similar messages]
W20240501 14:06:59.084133 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:06:59.207370 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:59.239537 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:59.305076 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:06:59.360095 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:06:59.473472 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240501 14:06:59.739436 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 14:06:59.749069 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:06:59.792307 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:06:59.832273 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:06:59.931916 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:07:00.113145 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:07:00.142292 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:07:00.198714 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:00.236712 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:00.351579 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:07:00.378336 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:07:00.420605 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:07:00.566094 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:07:00.577988 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:07:00.631860 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:00.746026 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:07:00.767186 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:07:00.777825 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:07:00.811328 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:00.847473 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:07:00.998332 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:07:01.018980 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:01.190253 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:07:01.209964 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 14:07:01.336410 17119 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:07:01.336699 17119 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:07:01.337647 17119 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers be56773105e744919601127a94e98adf (127.29.124.132:35459), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:07:01.337961 5386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf" is_pre_election: true
I20240501 14:07:01.338145 5386 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 2.
W20240501 14:07:01.338740 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:07:01.338995 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:01.339157 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:07:01.339218 5191 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:07:01.339421 17119 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:07:01.394098 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:01.567652 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:07:01.632570 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:01.648054 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:07:01.727532 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:07:01.821195 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:07:01.831748 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:02.027410 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:02.364933 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:02.440299 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:02.456346 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:07:02.481057 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:07:02.490407 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:07:02.540333 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:07:02.770756 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:07:02.882686 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:02.998761 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:03.004046 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:07:03.010542 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:07:03.035051 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:07:03.119474 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:07:03.368834 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:07:03.374248 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:07:03.459208 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:07:03.464867 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:03.537887 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:07:03.555564 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:07:03.604830 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:07:03.615690 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:03.814312 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:07:03.854835 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:03.901947 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:07:03.939237 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 14:07:04.086309 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:04.093789 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 67 similar messages]
W20240501 14:07:04.109623 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:07:04.111871 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:07:04.123646 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:07:04.391376 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:07:04.414022 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:04.533277 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 14:07:04.695079 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:07:04.761546 4843 maintenance_manager.cc:419] P f10c6a9c4bad46deb63190d64169ec11: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033367
W20240501 14:07:04.777827 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240501 14:07:04.794299 4842 maintenance_manager.cc:643] P f10c6a9c4bad46deb63190d64169ec11: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.032s user 0.022s sys 0.009s Metrics: {"bytes_written":20795,"cfile_init":1,"dirs.queue_time_us":323,"dirs.run_cpu_time_us":184,"dirs.run_wall_time_us":1058,"drs_written":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4,"lbm_write_time_us":974,"lbm_writes_lt_1ms":29,"peak_mem_usage":0,"rows_written":43,"thread_start_us":429,"threads_started":2,"wal-append.queue_time_us":290}
I20240501 14:07:04.795559 4843 maintenance_manager.cc:419] P f10c6a9c4bad46deb63190d64169ec11: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 10691 bytes on disk
I20240501 14:07:04.796284 4842 maintenance_manager.cc:643] P f10c6a9c4bad46deb63190d64169ec11: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
W20240501 14:07:04.823650 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:07:04.922293 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:07:04.932966 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:04.946682 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:07:05.206555 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:07:05.269357 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:07:05.315886 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:05.489352 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:07:05.535398 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:05.711005 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:07:05.776682 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:05.792407 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:07:05.825556 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:07:05.950090 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:07:06.062722 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:07:06.100278 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:07:06.105020 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:07:06.115664 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:07:06.144943 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:06.198585 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:07:06.244859 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:07:06.309113 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:06.352599 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:07:06.551957 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:07:06.569216 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:07:06.618862 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:06.687326 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:07:06.701743 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:07:06.750810 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:07.067809 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:07:07.137367 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:07.178956 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:07.357306 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:07:07.515177 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:07:07.522544 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:07:07.579872 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:07.668532 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:07.702977 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:07:07.963011 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:07:08.017154 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:07:08.022928 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:08.095620 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:07:08.124341 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:07:08.183708 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:08.277808 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:07:08.560914 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:08.567045 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:07:08.588260 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:07:08.643019 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:08.654973 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:07:08.662894 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:07:08.720294 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:07:08.851286 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:07:08.857622 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:07:08.927146 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:07:08.974699 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:07:09.008003 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 14:07:09.008873 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:09.151669 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240501 14:07:09.159409 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:07:09.206444 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:09.350998 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:07:09.448437 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:07:09.528450 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:09.588605 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240501 14:07:09.608466 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:09.740108 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:07:09.758285 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:07:09.928236 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:09.939046 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:07:09.985124 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:09.995719 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:07:10.121320 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:07:10.434666 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:07:10.506184 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:10.552765 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:10.567096 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:07:10.596493 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:07:10.672451 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:07:10.729456 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:07:10.782395 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:07:10.958910 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:10.995013 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:11.017096 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:07:11.084355 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:07:11.092310 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:07:11.278764 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:07:11.390965 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:07:11.408545 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:07:11.420318 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:07:11.480655 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:11.532868 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:07:11.541049 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:11.787089 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:07:11.824867 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:07:11.920035 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:11.968524 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:12.230085 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:07:12.294582 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:07:12.453217 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:12.457185 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:07:12.459515 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:12.546581 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:07:12.632162 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 14:07:12.773917 5302 maintenance_manager.cc:419] P 1c3fdc5272484ac09ef3e5e36a08f27c: Scheduling FlushMRSOp(88ed5438fc934adc9fb98c6cb4476598): perf score=0.033343
W20240501 14:07:12.780122 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 14:07:12.825914 5172 maintenance_manager.cc:419] P e9c60449cf3c46f8aca4ab6c535302a4: Scheduling FlushMRSOp(43b46fcf827c4275a47291afb107c290): perf score=0.033363
I20240501 14:07:12.870056 5193 maintenance_manager.cc:643] P 1c3fdc5272484ac09ef3e5e36a08f27c: FlushMRSOp(88ed5438fc934adc9fb98c6cb4476598) complete. Timing: real 0.095s user 0.055s sys 0.017s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":295,"dirs.queue_time_us":333,"dirs.run_cpu_time_us":218,"dirs.run_wall_time_us":1112,"drs_written":1,"lbm_read_time_us":117,"lbm_reads_lt_1ms":4,"lbm_write_time_us":829,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":7796,"threads_started":3,"wal-append.queue_time_us":6698}
I20240501 14:07:12.872476 5302 maintenance_manager.cc:419] P 1c3fdc5272484ac09ef3e5e36a08f27c: Scheduling UndoDeltaBlockGCOp(88ed5438fc934adc9fb98c6cb4476598): 4581 bytes on disk
I20240501 14:07:12.873612 5193 maintenance_manager.cc:643] P 1c3fdc5272484ac09ef3e5e36a08f27c: UndoDeltaBlockGCOp(88ed5438fc934adc9fb98c6cb4476598) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":193,"lbm_reads_lt_1ms":4}
W20240501 14:07:12.899778 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:07:12.932554 5063 maintenance_manager.cc:643] P e9c60449cf3c46f8aca4ab6c535302a4: FlushMRSOp(43b46fcf827c4275a47291afb107c290) complete. Timing: real 0.106s user 0.063s sys 0.022s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":389,"dirs.queue_time_us":377,"dirs.run_cpu_time_us":235,"dirs.run_wall_time_us":996,"drs_written":1,"lbm_read_time_us":109,"lbm_reads_lt_1ms":4,"lbm_write_time_us":950,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":751,"threads_started":3,"wal-append.queue_time_us":386}
I20240501 14:07:12.935032 5172 maintenance_manager.cc:419] P e9c60449cf3c46f8aca4ab6c535302a4: Scheduling FlushMRSOp(88ed5438fc934adc9fb98c6cb4476598): perf score=0.033387
W20240501 14:07:12.943758 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 14:07:12.966609 5432 maintenance_manager.cc:419] P be56773105e744919601127a94e98adf: Scheduling FlushMRSOp(43b46fcf827c4275a47291afb107c290): perf score=0.033396
W20240501 14:07:13.016036 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
I20240501 14:07:13.079896 5063 maintenance_manager.cc:643] P e9c60449cf3c46f8aca4ab6c535302a4: FlushMRSOp(88ed5438fc934adc9fb98c6cb4476598) complete. Timing: real 0.145s user 0.057s sys 0.032s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":57,"dirs.run_cpu_time_us":188,"dirs.run_wall_time_us":1760,"drs_written":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4,"lbm_write_time_us":878,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":34075,"threads_started":1,"wal-append.queue_time_us":30825}
I20240501 14:07:13.081336 5172 maintenance_manager.cc:419] P e9c60449cf3c46f8aca4ab6c535302a4: Scheduling UndoDeltaBlockGCOp(43b46fcf827c4275a47291afb107c290): 4833 bytes on disk
W20240501 14:07:13.096535 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 14:07:13.098775 5063 maintenance_manager.cc:643] P e9c60449cf3c46f8aca4ab6c535302a4: UndoDeltaBlockGCOp(43b46fcf827c4275a47291afb107c290) complete. Timing: real 0.003s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":121,"lbm_reads_lt_1ms":4}
I20240501 14:07:13.100678 18378 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:07:13.100777 5172 maintenance_manager.cc:419] P e9c60449cf3c46f8aca4ab6c535302a4: Scheduling UndoDeltaBlockGCOp(88ed5438fc934adc9fb98c6cb4476598): 4581 bytes on disk
I20240501 14:07:13.100840 18378 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:07:13.101807 18378 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:07:13.101785 5063 maintenance_manager.cc:643] P e9c60449cf3c46f8aca4ab6c535302a4: UndoDeltaBlockGCOp(88ed5438fc934adc9fb98c6cb4476598) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4}
I20240501 14:07:13.102690 5255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" is_pre_election: true
I20240501 14:07:13.102916 5255 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 2.
W20240501 14:07:13.105998 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:13.106320 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
I20240501 14:07:13.106470 5321 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:07:13.106722 18378 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:07:13.148772 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 14:07:13.188724 5323 maintenance_manager.cc:643] P be56773105e744919601127a94e98adf: FlushMRSOp(43b46fcf827c4275a47291afb107c290) complete. Timing: real 0.221s user 0.084s sys 0.027s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":14414,"dirs.queue_time_us":9965,"dirs.run_cpu_time_us":194,"dirs.run_wall_time_us":1096,"drs_written":1,"lbm_read_time_us":108,"lbm_reads_lt_1ms":4,"lbm_write_time_us":954,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":30129,"threads_started":3,"wal-append.queue_time_us":8951}
I20240501 14:07:13.190924 5432 maintenance_manager.cc:419] P be56773105e744919601127a94e98adf: Scheduling UndoDeltaBlockGCOp(43b46fcf827c4275a47291afb107c290): 4833 bytes on disk
I20240501 14:07:13.198212 5323 maintenance_manager.cc:643] P be56773105e744919601127a94e98adf: UndoDeltaBlockGCOp(43b46fcf827c4275a47291afb107c290) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":134,"lbm_reads_lt_1ms":4}
I20240501 14:07:13.199508 5432 maintenance_manager.cc:419] P be56773105e744919601127a94e98adf: Scheduling FlushMRSOp(9b88c43cb8704583a3e65c5c35c205f6): perf score=0.033456
W20240501 14:07:13.215489 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240501 14:07:13.282585 5323 maintenance_manager.cc:643] P be56773105e744919601127a94e98adf: FlushMRSOp(9b88c43cb8704583a3e65c5c35c205f6) complete. Timing: real 0.083s user 0.037s sys 0.023s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":60,"dirs.run_cpu_time_us":201,"dirs.run_wall_time_us":1465,"drs_written":1,"lbm_read_time_us":203,"lbm_reads_lt_1ms":4,"lbm_write_time_us":862,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":6647,"threads_started":1,"wal-append.queue_time_us":6528}
I20240501 14:07:13.284189 5432 maintenance_manager.cc:419] P be56773105e744919601127a94e98adf: Scheduling UndoDeltaBlockGCOp(9b88c43cb8704583a3e65c5c35c205f6): 4921 bytes on disk
I20240501 14:07:13.284996 5323 maintenance_manager.cc:643] P be56773105e744919601127a94e98adf: UndoDeltaBlockGCOp(9b88c43cb8704583a3e65c5c35c205f6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
W20240501 14:07:13.384948 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:07:13.425501 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:13.453225 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:07:13.512954 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:07:13.517688 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:13.566937 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:07:13.571915 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 14:07:13.787926 5432 maintenance_manager.cc:419] P be56773105e744919601127a94e98adf: Scheduling FlushMRSOp(f4d2b8c005b64bbb8e6621da068d15e5): perf score=0.033354
W20240501 14:07:13.812551 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 14:07:13.823189 5323 maintenance_manager.cc:643] P be56773105e744919601127a94e98adf: FlushMRSOp(f4d2b8c005b64bbb8e6621da068d15e5) complete. Timing: real 0.035s user 0.022s sys 0.011s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":867,"dirs.run_cpu_time_us":197,"dirs.run_wall_time_us":1645,"drs_written":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4,"lbm_write_time_us":862,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":1008,"threads_started":2,"wal-append.queue_time_us":338}
I20240501 14:07:13.824244 5432 maintenance_manager.cc:419] P be56773105e744919601127a94e98adf: Scheduling UndoDeltaBlockGCOp(f4d2b8c005b64bbb8e6621da068d15e5): 4581 bytes on disk
I20240501 14:07:13.824796 5323 maintenance_manager.cc:643] P be56773105e744919601127a94e98adf: UndoDeltaBlockGCOp(f4d2b8c005b64bbb8e6621da068d15e5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4}
I20240501 14:07:13.855479 5172 maintenance_manager.cc:419] P e9c60449cf3c46f8aca4ab6c535302a4: Scheduling FlushMRSOp(c8a961e711114f369ba06840c062b982): perf score=0.033369
I20240501 14:07:13.878429 5302 maintenance_manager.cc:419] P 1c3fdc5272484ac09ef3e5e36a08f27c: Scheduling FlushMRSOp(1a7bbfc15a76450bbc9197037e856e08): perf score=0.033377
I20240501 14:07:13.885906 5041 maintenance_manager.cc:419] P 04088ee3bea440a7bd5ff772152b9410: Scheduling FlushMRSOp(c8a961e711114f369ba06840c062b982): perf score=0.033383
W20240501 14:07:13.895879 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:13.907985 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:07:13.923305 5063 maintenance_manager.cc:643] P e9c60449cf3c46f8aca4ab6c535302a4: FlushMRSOp(c8a961e711114f369ba06840c062b982) complete. Timing: real 0.068s user 0.046s sys 0.014s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":358,"dirs.run_cpu_time_us":230,"dirs.run_wall_time_us":1503,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":710,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":12183,"threads_started":2,"wal-append.queue_time_us":12115}
I20240501 14:07:13.924674 5172 maintenance_manager.cc:419] P e9c60449cf3c46f8aca4ab6c535302a4: Scheduling UndoDeltaBlockGCOp(c8a961e711114f369ba06840c062b982): 4834 bytes on disk
I20240501 14:07:13.925393 5063 maintenance_manager.cc:643] P e9c60449cf3c46f8aca4ab6c535302a4: UndoDeltaBlockGCOp(c8a961e711114f369ba06840c062b982) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":109,"lbm_reads_lt_1ms":4}
I20240501 14:07:13.934338 5193 maintenance_manager.cc:643] P 1c3fdc5272484ac09ef3e5e36a08f27c: FlushMRSOp(1a7bbfc15a76450bbc9197037e856e08) complete. Timing: real 0.056s user 0.039s sys 0.010s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":7134,"dirs.run_cpu_time_us":253,"dirs.run_wall_time_us":10366,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1773,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":7639,"threads_started":2,"wal-append.queue_time_us":412}
I20240501 14:07:13.935698 5302 maintenance_manager.cc:419] P 1c3fdc5272484ac09ef3e5e36a08f27c: Scheduling UndoDeltaBlockGCOp(1a7bbfc15a76450bbc9197037e856e08): 4921 bytes on disk
I20240501 14:07:13.936266 5193 maintenance_manager.cc:643] P 1c3fdc5272484ac09ef3e5e36a08f27c: UndoDeltaBlockGCOp(1a7bbfc15a76450bbc9197037e856e08) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4}
I20240501 14:07:13.969653 4932 maintenance_manager.cc:643] P 04088ee3bea440a7bd5ff772152b9410: FlushMRSOp(c8a961e711114f369ba06840c062b982) complete. Timing: real 0.083s user 0.062s sys 0.011s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":355,"dirs.queue_time_us":427,"dirs.run_cpu_time_us":242,"dirs.run_wall_time_us":905,"drs_written":1,"lbm_read_time_us":109,"lbm_reads_lt_1ms":4,"lbm_write_time_us":980,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":775,"threads_started":3,"wal-append.queue_time_us":378}
I20240501 14:07:13.971706 5041 maintenance_manager.cc:419] P 04088ee3bea440a7bd5ff772152b9410: Scheduling UndoDeltaBlockGCOp(c8a961e711114f369ba06840c062b982): 4834 bytes on disk
I20240501 14:07:13.972843 4932 maintenance_manager.cc:643] P 04088ee3bea440a7bd5ff772152b9410: UndoDeltaBlockGCOp(c8a961e711114f369ba06840c062b982) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":165,"lbm_reads_lt_1ms":4}
W20240501 14:07:13.975374 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:07:14.016098 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:07:14.081194 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 14:07:14.109377 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:07:14.207834 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240501 14:07:14.299742 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:14.347860 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:07:14.356745 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:14.423017 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:07:14.521054 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:07:14.629334 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 14:07:14.785915 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:14.885039 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:14.925101 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 14:07:14.930141 5172 maintenance_manager.cc:419] P e9c60449cf3c46f8aca4ab6c535302a4: Scheduling FlushMRSOp(a726769d24284fedadd8e9e63cd27d05): perf score=0.033338
W20240501 14:07:14.938198 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 14:07:14.940948 5302 maintenance_manager.cc:419] P 1c3fdc5272484ac09ef3e5e36a08f27c: Scheduling FlushMRSOp(71660575d94748d8a9329f909a42a0cd): perf score=0.033337
W20240501 14:07:14.957824 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240501 14:07:14.977695 5041 maintenance_manager.cc:419] P 04088ee3bea440a7bd5ff772152b9410: Scheduling FlushMRSOp(71660575d94748d8a9329f909a42a0cd): perf score=0.033343
I20240501 14:07:14.997778 5063 maintenance_manager.cc:643] P e9c60449cf3c46f8aca4ab6c535302a4: FlushMRSOp(a726769d24284fedadd8e9e63cd27d05) complete. Timing: real 0.067s user 0.032s sys 0.029s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":6463,"dirs.run_cpu_time_us":237,"dirs.run_wall_time_us":9414,"drs_written":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4,"lbm_write_time_us":923,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":14395,"threads_started":2,"wal-append.queue_time_us":7873}
I20240501 14:07:14.999168 5172 maintenance_manager.cc:419] P e9c60449cf3c46f8aca4ab6c535302a4: Scheduling UndoDeltaBlockGCOp(a726769d24284fedadd8e9e63cd27d05): 4921 bytes on disk
I20240501 14:07:14.999900 5063 maintenance_manager.cc:643] P e9c60449cf3c46f8aca4ab6c535302a4: UndoDeltaBlockGCOp(a726769d24284fedadd8e9e63cd27d05) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":119,"lbm_reads_lt_1ms":4}
I20240501 14:07:15.000794 5172 maintenance_manager.cc:419] P e9c60449cf3c46f8aca4ab6c535302a4: Scheduling FlushMRSOp(fee16905387d428aa02cd2070c86c790): perf score=0.033351
I20240501 14:07:15.007970 5193 maintenance_manager.cc:643] P 1c3fdc5272484ac09ef3e5e36a08f27c: FlushMRSOp(71660575d94748d8a9329f909a42a0cd) complete. Timing: real 0.067s user 0.044s sys 0.014s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":375,"dirs.run_cpu_time_us":186,"dirs.run_wall_time_us":1315,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":890,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":1034,"threads_started":2,"wal-append.queue_time_us":387}
I20240501 14:07:15.009393 5302 maintenance_manager.cc:419] P 1c3fdc5272484ac09ef3e5e36a08f27c: Scheduling UndoDeltaBlockGCOp(71660575d94748d8a9329f909a42a0cd): 4581 bytes on disk
I20240501 14:07:15.010051 5193 maintenance_manager.cc:643] P 1c3fdc5272484ac09ef3e5e36a08f27c: UndoDeltaBlockGCOp(71660575d94748d8a9329f909a42a0cd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4}
W20240501 14:07:15.012948 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 14:07:15.024452 4932 maintenance_manager.cc:643] P 04088ee3bea440a7bd5ff772152b9410: FlushMRSOp(71660575d94748d8a9329f909a42a0cd) complete. Timing: real 0.046s user 0.029s sys 0.011s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":1480,"dirs.run_cpu_time_us":235,"dirs.run_wall_time_us":1441,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":711,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":7758,"threads_started":2,"wal-append.queue_time_us":6281}
I20240501 14:07:15.025524 5041 maintenance_manager.cc:419] P 04088ee3bea440a7bd5ff772152b9410: Scheduling FlushMRSOp(fee16905387d428aa02cd2070c86c790): perf score=0.033355
W20240501 14:07:15.038328 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 41 similar messages]
I20240501 14:07:15.039647 5063 maintenance_manager.cc:643] P e9c60449cf3c46f8aca4ab6c535302a4: FlushMRSOp(fee16905387d428aa02cd2070c86c790) complete. Timing: real 0.039s user 0.030s sys 0.007s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":55,"dirs.run_cpu_time_us":186,"dirs.run_wall_time_us":1251,"drs_written":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4,"lbm_write_time_us":664,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":265,"threads_started":1,"wal-append.queue_time_us":371}
I20240501 14:07:15.040834 5172 maintenance_manager.cc:419] P e9c60449cf3c46f8aca4ab6c535302a4: Scheduling UndoDeltaBlockGCOp(fee16905387d428aa02cd2070c86c790): 4834 bytes on disk
I20240501 14:07:15.041467 5063 maintenance_manager.cc:643] P e9c60449cf3c46f8aca4ab6c535302a4: UndoDeltaBlockGCOp(fee16905387d428aa02cd2070c86c790) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4}
I20240501 14:07:15.061691 4932 maintenance_manager.cc:643] P 04088ee3bea440a7bd5ff772152b9410: FlushMRSOp(fee16905387d428aa02cd2070c86c790) complete. Timing: real 0.036s user 0.026s sys 0.008s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":58,"dirs.run_cpu_time_us":208,"dirs.run_wall_time_us":1459,"drs_written":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4,"lbm_write_time_us":606,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":199,"threads_started":1,"wal-append.queue_time_us":304}
I20240501 14:07:15.062705 5041 maintenance_manager.cc:419] P 04088ee3bea440a7bd5ff772152b9410: Scheduling UndoDeltaBlockGCOp(fee16905387d428aa02cd2070c86c790): 4834 bytes on disk
I20240501 14:07:15.063207 4932 maintenance_manager.cc:643] P 04088ee3bea440a7bd5ff772152b9410: UndoDeltaBlockGCOp(fee16905387d428aa02cd2070c86c790) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4}
I20240501 14:07:15.063995 5041 maintenance_manager.cc:419] P 04088ee3bea440a7bd5ff772152b9410: Scheduling UndoDeltaBlockGCOp(71660575d94748d8a9329f909a42a0cd): 4581 bytes on disk
I20240501 14:07:15.064492 4932 maintenance_manager.cc:643] P 04088ee3bea440a7bd5ff772152b9410: UndoDeltaBlockGCOp(71660575d94748d8a9329f909a42a0cd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
I20240501 14:07:15.079926 5432 maintenance_manager.cc:419] P be56773105e744919601127a94e98adf: Scheduling FlushMRSOp(a726769d24284fedadd8e9e63cd27d05): perf score=0.033375
I20240501 14:07:15.116488 5323 maintenance_manager.cc:643] P be56773105e744919601127a94e98adf: FlushMRSOp(a726769d24284fedadd8e9e63cd27d05) complete. Timing: real 0.036s user 0.009s sys 0.024s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":333,"dirs.run_cpu_time_us":259,"dirs.run_wall_time_us":1532,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":587,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":1044,"threads_started":2,"wal-append.queue_time_us":1193}
I20240501 14:07:15.117617 5432 maintenance_manager.cc:419] P be56773105e744919601127a94e98adf: Scheduling UndoDeltaBlockGCOp(a726769d24284fedadd8e9e63cd27d05): 4921 bytes on disk
I20240501 14:07:15.118218 5323 maintenance_manager.cc:643] P be56773105e744919601127a94e98adf: UndoDeltaBlockGCOp(a726769d24284fedadd8e9e63cd27d05) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4}
W20240501 14:07:15.314904 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:07:15.321849 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:15.322978 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:15.483961 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:07:15.669618 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:07:15.671713 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:07:15.862383 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:15.876969 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:07:15.878105 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:07:15.881006 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:15.968024 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:07:16.030557 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 14:07:16.045248 5172 maintenance_manager.cc:419] P e9c60449cf3c46f8aca4ab6c535302a4: Scheduling FlushMRSOp(9afe3b20a43d4b5bb800af39f945948d): perf score=0.033336
I20240501 14:07:16.068809 5041 maintenance_manager.cc:419] P 04088ee3bea440a7bd5ff772152b9410: Scheduling FlushMRSOp(9afe3b20a43d4b5bb800af39f945948d): perf score=0.033348
I20240501 14:07:16.085619 5063 maintenance_manager.cc:643] P e9c60449cf3c46f8aca4ab6c535302a4: FlushMRSOp(9afe3b20a43d4b5bb800af39f945948d) complete. Timing: real 0.040s user 0.034s sys 0.006s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":387,"dirs.run_cpu_time_us":222,"dirs.run_wall_time_us":1627,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":651,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":456,"threads_started":2,"wal-append.queue_time_us":312}
I20240501 14:07:16.086659 5172 maintenance_manager.cc:419] P e9c60449cf3c46f8aca4ab6c535302a4: Scheduling FlushMRSOp(404e2851055541f0be1bb0971a7cc72a): perf score=0.033345
I20240501 14:07:16.119515 4932 maintenance_manager.cc:643] P 04088ee3bea440a7bd5ff772152b9410: FlushMRSOp(9afe3b20a43d4b5bb800af39f945948d) complete. Timing: real 0.050s user 0.031s sys 0.016s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":331,"dirs.run_cpu_time_us":222,"dirs.run_wall_time_us":1552,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":656,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":437,"threads_started":2,"wal-append.queue_time_us":316}
I20240501 14:07:16.121001 5041 maintenance_manager.cc:419] P 04088ee3bea440a7bd5ff772152b9410: Scheduling UndoDeltaBlockGCOp(9afe3b20a43d4b5bb800af39f945948d): 4921 bytes on disk
I20240501 14:07:16.121811 4932 maintenance_manager.cc:643] P 04088ee3bea440a7bd5ff772152b9410: UndoDeltaBlockGCOp(9afe3b20a43d4b5bb800af39f945948d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":111,"lbm_reads_lt_1ms":4}
I20240501 14:07:16.122849 5432 maintenance_manager.cc:419] P be56773105e744919601127a94e98adf: Scheduling FlushMRSOp(51cbcf4888c440c2ad096c988ab2d968): perf score=0.033362
I20240501 14:07:16.126956 5063 maintenance_manager.cc:643] P e9c60449cf3c46f8aca4ab6c535302a4: FlushMRSOp(404e2851055541f0be1bb0971a7cc72a) complete. Timing: real 0.040s user 0.026s sys 0.012s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":90,"dirs.run_cpu_time_us":197,"dirs.run_wall_time_us":1638,"drs_written":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4,"lbm_write_time_us":801,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":1489,"threads_started":1,"wal-append.queue_time_us":1577}
I20240501 14:07:16.128458 5172 maintenance_manager.cc:419] P e9c60449cf3c46f8aca4ab6c535302a4: Scheduling UndoDeltaBlockGCOp(9afe3b20a43d4b5bb800af39f945948d): 4921 bytes on disk
I20240501 14:07:16.129118 5063 maintenance_manager.cc:643] P e9c60449cf3c46f8aca4ab6c535302a4: UndoDeltaBlockGCOp(9afe3b20a43d4b5bb800af39f945948d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":112,"lbm_reads_lt_1ms":4}
I20240501 14:07:16.130264 5172 maintenance_manager.cc:419] P e9c60449cf3c46f8aca4ab6c535302a4: Scheduling UndoDeltaBlockGCOp(404e2851055541f0be1bb0971a7cc72a): 4834 bytes on disk
I20240501 14:07:16.130821 5063 maintenance_manager.cc:643] P e9c60449cf3c46f8aca4ab6c535302a4: UndoDeltaBlockGCOp(404e2851055541f0be1bb0971a7cc72a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4}
I20240501 14:07:16.160861 5323 maintenance_manager.cc:643] P be56773105e744919601127a94e98adf: FlushMRSOp(51cbcf4888c440c2ad096c988ab2d968) complete. Timing: real 0.038s user 0.025s sys 0.008s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":319,"dirs.run_cpu_time_us":194,"dirs.run_wall_time_us":1392,"drs_written":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4,"lbm_write_time_us":671,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":428,"threads_started":2,"wal-append.queue_time_us":308}
I20240501 14:07:16.162178 5432 maintenance_manager.cc:419] P be56773105e744919601127a94e98adf: Scheduling UndoDeltaBlockGCOp(51cbcf4888c440c2ad096c988ab2d968): 4581 bytes on disk
I20240501 14:07:16.162719 5323 maintenance_manager.cc:643] P be56773105e744919601127a94e98adf: UndoDeltaBlockGCOp(51cbcf4888c440c2ad096c988ab2d968) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4}
W20240501 14:07:16.225999 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 14:07:16.264546 5302 maintenance_manager.cc:419] P 1c3fdc5272484ac09ef3e5e36a08f27c: Scheduling FlushMRSOp(51cbcf4888c440c2ad096c988ab2d968): perf score=0.033392
W20240501 14:07:16.283689 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240501 14:07:16.300374 5193 maintenance_manager.cc:643] P 1c3fdc5272484ac09ef3e5e36a08f27c: FlushMRSOp(51cbcf4888c440c2ad096c988ab2d968) complete. Timing: real 0.036s user 0.029s sys 0.003s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":289,"dirs.run_cpu_time_us":201,"dirs.run_wall_time_us":1315,"drs_written":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4,"lbm_write_time_us":547,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":425,"threads_started":2,"wal-append.queue_time_us":310}
I20240501 14:07:16.301563 5302 maintenance_manager.cc:419] P 1c3fdc5272484ac09ef3e5e36a08f27c: Scheduling FlushMRSOp(404e2851055541f0be1bb0971a7cc72a): perf score=0.033401
W20240501 14:07:16.313921 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:16.333798 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 14:07:16.334369 5193 maintenance_manager.cc:643] P 1c3fdc5272484ac09ef3e5e36a08f27c: FlushMRSOp(404e2851055541f0be1bb0971a7cc72a) complete. Timing: real 0.033s user 0.019s sys 0.012s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":47,"dirs.run_cpu_time_us":184,"dirs.run_wall_time_us":1321,"drs_written":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4,"lbm_write_time_us":560,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":225,"threads_started":1,"wal-append.queue_time_us":323}
I20240501 14:07:16.335356 5302 maintenance_manager.cc:419] P 1c3fdc5272484ac09ef3e5e36a08f27c: Scheduling UndoDeltaBlockGCOp(404e2851055541f0be1bb0971a7cc72a): 4834 bytes on disk
I20240501 14:07:16.335839 5193 maintenance_manager.cc:643] P 1c3fdc5272484ac09ef3e5e36a08f27c: UndoDeltaBlockGCOp(404e2851055541f0be1bb0971a7cc72a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
I20240501 14:07:16.336632 5302 maintenance_manager.cc:419] P 1c3fdc5272484ac09ef3e5e36a08f27c: Scheduling UndoDeltaBlockGCOp(51cbcf4888c440c2ad096c988ab2d968): 4581 bytes on disk
I20240501 14:07:16.337060 5193 maintenance_manager.cc:643] P 1c3fdc5272484ac09ef3e5e36a08f27c: UndoDeltaBlockGCOp(51cbcf4888c440c2ad096c988ab2d968) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4}
W20240501 14:07:16.434760 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:16.611560 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:07:16.675253 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:07:16.716584 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:07:16.810892 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:16.816071 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:07:16.910352 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:16.965961 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:07:17.105612 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:07:17.131302 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:07:17.197925 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:07:17.335785 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:17.391042 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:17.412613 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:07:17.432837 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:07:17.608353 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:07:17.835655 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:17.860430 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:18.015152 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 14:07:18.092995 5302 maintenance_manager.cc:419] P 1c3fdc5272484ac09ef3e5e36a08f27c: Scheduling FlushMRSOp(ba6cba753e934e719c730593c0706346): perf score=0.033349
W20240501 14:07:18.117686 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 14:07:18.129348 5041 maintenance_manager.cc:419] P 04088ee3bea440a7bd5ff772152b9410: Scheduling FlushMRSOp(ba6cba753e934e719c730593c0706346): perf score=0.033364
I20240501 14:07:18.138464 5172 maintenance_manager.cc:419] P e9c60449cf3c46f8aca4ab6c535302a4: Scheduling FlushMRSOp(b284aa82594a44f5885f32fa05844913): perf score=0.033364
I20240501 14:07:18.142571 5193 maintenance_manager.cc:643] P 1c3fdc5272484ac09ef3e5e36a08f27c: FlushMRSOp(ba6cba753e934e719c730593c0706346) complete. Timing: real 0.049s user 0.035s sys 0.013s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":982,"dirs.run_cpu_time_us":225,"dirs.run_wall_time_us":1548,"drs_written":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4,"lbm_write_time_us":664,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":1179,"threads_started":2,"wal-append.queue_time_us":707}
I20240501 14:07:18.143834 5302 maintenance_manager.cc:419] P 1c3fdc5272484ac09ef3e5e36a08f27c: Scheduling UndoDeltaBlockGCOp(ba6cba753e934e719c730593c0706346): 4921 bytes on disk
I20240501 14:07:18.144321 5193 maintenance_manager.cc:643] P 1c3fdc5272484ac09ef3e5e36a08f27c: UndoDeltaBlockGCOp(ba6cba753e934e719c730593c0706346) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
I20240501 14:07:18.165329 4932 maintenance_manager.cc:643] P 04088ee3bea440a7bd5ff772152b9410: FlushMRSOp(ba6cba753e934e719c730593c0706346) complete. Timing: real 0.036s user 0.017s sys 0.017s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":367,"dirs.run_cpu_time_us":210,"dirs.run_wall_time_us":1288,"drs_written":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4,"lbm_write_time_us":549,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":452,"threads_started":2,"wal-append.queue_time_us":281}
I20240501 14:07:18.166373 5041 maintenance_manager.cc:419] P 04088ee3bea440a7bd5ff772152b9410: Scheduling UndoDeltaBlockGCOp(ba6cba753e934e719c730593c0706346): 4921 bytes on disk
I20240501 14:07:18.166918 4932 maintenance_manager.cc:643] P 04088ee3bea440a7bd5ff772152b9410: UndoDeltaBlockGCOp(ba6cba753e934e719c730593c0706346) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4}
I20240501 14:07:18.167922 5041 maintenance_manager.cc:419] P 04088ee3bea440a7bd5ff772152b9410: Scheduling FlushMRSOp(b284aa82594a44f5885f32fa05844913): perf score=0.033373
I20240501 14:07:18.170230 5432 maintenance_manager.cc:419] P be56773105e744919601127a94e98adf: Scheduling FlushMRSOp(b284aa82594a44f5885f32fa05844913): perf score=0.033375
I20240501 14:07:18.170140 5063 maintenance_manager.cc:643] P e9c60449cf3c46f8aca4ab6c535302a4: FlushMRSOp(b284aa82594a44f5885f32fa05844913) complete. Timing: real 0.031s user 0.019s sys 0.011s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":299,"dirs.run_cpu_time_us":215,"dirs.run_wall_time_us":1182,"drs_written":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4,"lbm_write_time_us":686,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":383,"threads_started":2,"wal-append.queue_time_us":274}
I20240501 14:07:18.171370 5172 maintenance_manager.cc:419] P e9c60449cf3c46f8aca4ab6c535302a4: Scheduling UndoDeltaBlockGCOp(b284aa82594a44f5885f32fa05844913): 4581 bytes on disk
I20240501 14:07:18.171955 5063 maintenance_manager.cc:643] P e9c60449cf3c46f8aca4ab6c535302a4: UndoDeltaBlockGCOp(b284aa82594a44f5885f32fa05844913) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4}
I20240501 14:07:18.211190 4932 maintenance_manager.cc:643] P 04088ee3bea440a7bd5ff772152b9410: FlushMRSOp(b284aa82594a44f5885f32fa05844913) complete. Timing: real 0.043s user 0.034s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":36,"dirs.run_cpu_time_us":162,"dirs.run_wall_time_us":1225,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":712,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":207,"threads_started":1,"wal-append.queue_time_us":298}
I20240501 14:07:18.212121 5041 maintenance_manager.cc:419] P 04088ee3bea440a7bd5ff772152b9410: Scheduling UndoDeltaBlockGCOp(b284aa82594a44f5885f32fa05844913): 4581 bytes on disk
I20240501 14:07:18.212612 4932 maintenance_manager.cc:643] P 04088ee3bea440a7bd5ff772152b9410: UndoDeltaBlockGCOp(b284aa82594a44f5885f32fa05844913) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4}
I20240501 14:07:18.214359 5323 maintenance_manager.cc:643] P be56773105e744919601127a94e98adf: FlushMRSOp(b284aa82594a44f5885f32fa05844913) complete. Timing: real 0.044s user 0.033s sys 0.008s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":311,"dirs.run_cpu_time_us":198,"dirs.run_wall_time_us":2938,"drs_written":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4,"lbm_write_time_us":714,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":6571,"threads_started":2,"wal-append.queue_time_us":6267}
I20240501 14:07:18.215617 5432 maintenance_manager.cc:419] P be56773105e744919601127a94e98adf: Scheduling UndoDeltaBlockGCOp(b284aa82594a44f5885f32fa05844913): 4581 bytes on disk
I20240501 14:07:18.216210 5323 maintenance_manager.cc:643] P be56773105e744919601127a94e98adf: UndoDeltaBlockGCOp(b284aa82594a44f5885f32fa05844913) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4}
I20240501 14:07:18.217058 5432 maintenance_manager.cc:419] P be56773105e744919601127a94e98adf: Scheduling FlushMRSOp(e7d5f02ccba34254a4d20951878b1214): perf score=0.033384
W20240501 14:07:18.227252 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:07:18.269042 5323 maintenance_manager.cc:643] P be56773105e744919601127a94e98adf: FlushMRSOp(e7d5f02ccba34254a4d20951878b1214) complete. Timing: real 0.052s user 0.034s sys 0.015s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":48,"dirs.run_cpu_time_us":198,"dirs.run_wall_time_us":1454,"drs_written":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4,"lbm_write_time_us":582,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":199,"threads_started":1,"wal-append.queue_time_us":296}
I20240501 14:07:18.270076 5432 maintenance_manager.cc:419] P be56773105e744919601127a94e98adf: Scheduling UndoDeltaBlockGCOp(e7d5f02ccba34254a4d20951878b1214): 4834 bytes on disk
I20240501 14:07:18.270592 5323 maintenance_manager.cc:643] P be56773105e744919601127a94e98adf: UndoDeltaBlockGCOp(e7d5f02ccba34254a4d20951878b1214) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
W20240501 14:07:18.277006 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:07:18.283628 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:07:18.324189 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:07:18.407261 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:07:18.425035 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:18.464745 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:07:18.493136 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:07:18.844291 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:18.937366 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:18.949520 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:07:18.962867 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:07:19.050361 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 14:07:19.148802 5302 maintenance_manager.cc:419] P 1c3fdc5272484ac09ef3e5e36a08f27c: Scheduling FlushMRSOp(9fd94635797d447cbdc9181ee91f465b): perf score=0.033342
I20240501 14:07:19.175974 5172 maintenance_manager.cc:419] P e9c60449cf3c46f8aca4ab6c535302a4: Scheduling FlushMRSOp(f9cb784f60774ea7b6e70ccfebf867a8): perf score=0.033348
I20240501 14:07:19.178629 5193 maintenance_manager.cc:643] P 1c3fdc5272484ac09ef3e5e36a08f27c: FlushMRSOp(9fd94635797d447cbdc9181ee91f465b) complete. Timing: real 0.030s user 0.024s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":304,"dirs.run_cpu_time_us":181,"dirs.run_wall_time_us":1486,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":575,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":386,"threads_started":2,"wal-append.queue_time_us":295}
I20240501 14:07:19.179689 5302 maintenance_manager.cc:419] P 1c3fdc5272484ac09ef3e5e36a08f27c: Scheduling UndoDeltaBlockGCOp(9fd94635797d447cbdc9181ee91f465b): 4834 bytes on disk
I20240501 14:07:19.180248 5193 maintenance_manager.cc:643] P 1c3fdc5272484ac09ef3e5e36a08f27c: UndoDeltaBlockGCOp(9fd94635797d447cbdc9181ee91f465b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4}
I20240501 14:07:19.216586 5041 maintenance_manager.cc:419] P 04088ee3bea440a7bd5ff772152b9410: Scheduling FlushMRSOp(9fd94635797d447cbdc9181ee91f465b): perf score=0.033363
W20240501 14:07:19.221397 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 68 similar messages]
I20240501 14:07:19.224830 5063 maintenance_manager.cc:643] P e9c60449cf3c46f8aca4ab6c535302a4: FlushMRSOp(f9cb784f60774ea7b6e70ccfebf867a8) complete. Timing: real 0.049s user 0.030s sys 0.015s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":364,"dirs.run_cpu_time_us":245,"dirs.run_wall_time_us":1614,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":736,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":737,"threads_started":2,"wal-append.queue_time_us":571}
I20240501 14:07:19.225852 5172 maintenance_manager.cc:419] P e9c60449cf3c46f8aca4ab6c535302a4: Scheduling UndoDeltaBlockGCOp(f9cb784f60774ea7b6e70ccfebf867a8): 4921 bytes on disk
I20240501 14:07:19.226366 5063 maintenance_manager.cc:643] P e9c60449cf3c46f8aca4ab6c535302a4: UndoDeltaBlockGCOp(f9cb784f60774ea7b6e70ccfebf867a8) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
W20240501 14:07:19.242369 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 14:07:19.248522 4932 maintenance_manager.cc:643] P 04088ee3bea440a7bd5ff772152b9410: FlushMRSOp(9fd94635797d447cbdc9181ee91f465b) complete. Timing: real 0.032s user 0.019s sys 0.012s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":312,"dirs.run_cpu_time_us":172,"dirs.run_wall_time_us":1280,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":524,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":441,"threads_started":2,"wal-append.queue_time_us":327}
I20240501 14:07:19.249714 5041 maintenance_manager.cc:419] P 04088ee3bea440a7bd5ff772152b9410: Scheduling UndoDeltaBlockGCOp(9fd94635797d447cbdc9181ee91f465b): 4834 bytes on disk
W20240501 14:07:19.250033 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 61 similar messages]
I20240501 14:07:19.250237 4932 maintenance_manager.cc:643] P 04088ee3bea440a7bd5ff772152b9410: UndoDeltaBlockGCOp(9fd94635797d447cbdc9181ee91f465b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
I20240501 14:07:19.274389 5432 maintenance_manager.cc:419] P be56773105e744919601127a94e98adf: Scheduling FlushMRSOp(2c39b53867e047faaf24cc6f87a8100c): perf score=0.033372
W20240501 14:07:19.281072 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:07:19.285748 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 14:07:19.303629 5323 maintenance_manager.cc:643] P be56773105e744919601127a94e98adf: FlushMRSOp(2c39b53867e047faaf24cc6f87a8100c) complete. Timing: real 0.029s user 0.018s sys 0.009s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":317,"dirs.run_cpu_time_us":229,"dirs.run_wall_time_us":1526,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":558,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":409,"threads_started":2,"wal-append.queue_time_us":299}
I20240501 14:07:19.304706 5432 maintenance_manager.cc:419] P be56773105e744919601127a94e98adf: Scheduling UndoDeltaBlockGCOp(2c39b53867e047faaf24cc6f87a8100c): 4581 bytes on disk
I20240501 14:07:19.305212 5323 maintenance_manager.cc:643] P be56773105e744919601127a94e98adf: UndoDeltaBlockGCOp(2c39b53867e047faaf24cc6f87a8100c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4}
W20240501 14:07:19.391825 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:07:19.440282 18904 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:07:19.440440 18904 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:07:19.441130 18904 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:07:19.441567 5255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" is_pre_election: true
I20240501 14:07:19.441790 5255 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 2.
W20240501 14:07:19.442637 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:19.442795 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:07:19.442865 5321 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:07:19.443058 18904 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:07:19.490685 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:19.545051 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:07:19.641486 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:07:19.642128 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:07:19.650707 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 14:07:19.651654 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:07:19.827116 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:07:19.945801 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:20.065027 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:07:20.090826 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:20.108592 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:07:20.130508 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 40 similar messages]
I20240501 14:07:20.184165 5302 maintenance_manager.cc:419] P 1c3fdc5272484ac09ef3e5e36a08f27c: Scheduling FlushMRSOp(de6c7753c8564107b858876fcfa5924d): perf score=0.033337
I20240501 14:07:20.223770 5193 maintenance_manager.cc:643] P 1c3fdc5272484ac09ef3e5e36a08f27c: FlushMRSOp(de6c7753c8564107b858876fcfa5924d) complete. Timing: real 0.039s user 0.026s sys 0.011s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":313,"dirs.run_cpu_time_us":249,"dirs.run_wall_time_us":1460,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":598,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":412,"threads_started":2,"wal-append.queue_time_us":305}
I20240501 14:07:20.224774 5302 maintenance_manager.cc:419] P 1c3fdc5272484ac09ef3e5e36a08f27c: Scheduling UndoDeltaBlockGCOp(de6c7753c8564107b858876fcfa5924d): 4581 bytes on disk
I20240501 14:07:20.225386 5193 maintenance_manager.cc:643] P 1c3fdc5272484ac09ef3e5e36a08f27c: UndoDeltaBlockGCOp(de6c7753c8564107b858876fcfa5924d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":138,"lbm_reads_lt_1ms":4}
I20240501 14:07:20.229821 5172 maintenance_manager.cc:419] P e9c60449cf3c46f8aca4ab6c535302a4: Scheduling FlushMRSOp(ac4b234f794149e4b6def7a3b358942d): perf score=0.033347
I20240501 14:07:20.254252 5041 maintenance_manager.cc:419] P 04088ee3bea440a7bd5ff772152b9410: Scheduling FlushMRSOp(8cc5eec4600640f7a293877c825f8d96): perf score=0.033358
I20240501 14:07:20.262624 5063 maintenance_manager.cc:643] P e9c60449cf3c46f8aca4ab6c535302a4: FlushMRSOp(ac4b234f794149e4b6def7a3b358942d) complete. Timing: real 0.033s user 0.023s sys 0.008s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":295,"dirs.run_cpu_time_us":183,"dirs.run_wall_time_us":1461,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":573,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":406,"threads_started":2,"wal-append.queue_time_us":303}
I20240501 14:07:20.263988 5172 maintenance_manager.cc:419] P e9c60449cf3c46f8aca4ab6c535302a4: Scheduling UndoDeltaBlockGCOp(ac4b234f794149e4b6def7a3b358942d): 4921 bytes on disk
I20240501 14:07:20.264642 5063 maintenance_manager.cc:643] P e9c60449cf3c46f8aca4ab6c535302a4: UndoDeltaBlockGCOp(ac4b234f794149e4b6def7a3b358942d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4}
I20240501 14:07:20.298344 4932 maintenance_manager.cc:643] P 04088ee3bea440a7bd5ff772152b9410: FlushMRSOp(8cc5eec4600640f7a293877c825f8d96) complete. Timing: real 0.044s user 0.039s sys 0.001s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":311,"dirs.run_cpu_time_us":190,"dirs.run_wall_time_us":1501,"drs_written":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4,"lbm_write_time_us":629,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":425,"threads_started":2,"wal-append.queue_time_us":421}
I20240501 14:07:20.299276 5041 maintenance_manager.cc:419] P 04088ee3bea440a7bd5ff772152b9410: Scheduling UndoDeltaBlockGCOp(8cc5eec4600640f7a293877c825f8d96): 4834 bytes on disk
I20240501 14:07:20.299736 4932 maintenance_manager.cc:643] P 04088ee3bea440a7bd5ff772152b9410: UndoDeltaBlockGCOp(8cc5eec4600640f7a293877c825f8d96) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
W20240501 14:07:20.439339 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:07:20.497049 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:20.497908 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:20.674592 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:07:20.722100 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:07:20.777060 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:07:20.866834 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:07:20.875946 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:07:20.890784 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:20.929625 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:07:21.016064 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:21.037586 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:07:21.317370 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:21.339589 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:07:21.438759 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240501 14:07:21.479998 5302 maintenance_manager.cc:419] P 1c3fdc5272484ac09ef3e5e36a08f27c: Scheduling FlushMRSOp(63650547ab444f45bb377bcb49f258ad): perf score=0.033344
I20240501 14:07:21.511425 5193 maintenance_manager.cc:643] P 1c3fdc5272484ac09ef3e5e36a08f27c: FlushMRSOp(63650547ab444f45bb377bcb49f258ad) complete. Timing: real 0.031s user 0.011s sys 0.018s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":310,"dirs.run_cpu_time_us":216,"dirs.run_wall_time_us":1465,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":592,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":427,"threads_started":2,"wal-append.queue_time_us":311}
I20240501 14:07:21.512415 5302 maintenance_manager.cc:419] P 1c3fdc5272484ac09ef3e5e36a08f27c: Scheduling UndoDeltaBlockGCOp(63650547ab444f45bb377bcb49f258ad): 4921 bytes on disk
I20240501 14:07:21.512872 5193 maintenance_manager.cc:643] P 1c3fdc5272484ac09ef3e5e36a08f27c: UndoDeltaBlockGCOp(63650547ab444f45bb377bcb49f258ad) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
I20240501 14:07:21.513729 5302 maintenance_manager.cc:419] P 1c3fdc5272484ac09ef3e5e36a08f27c: Scheduling FlushMRSOp(23ca2bf1e59f44fcac8a48aff8d056b1): perf score=0.033349
W20240501 14:07:21.536924 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 14:07:21.544344 5193 maintenance_manager.cc:643] P 1c3fdc5272484ac09ef3e5e36a08f27c: FlushMRSOp(23ca2bf1e59f44fcac8a48aff8d056b1) complete. Timing: real 0.030s user 0.028s sys 0.000s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":44,"dirs.run_cpu_time_us":132,"dirs.run_wall_time_us":1459,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":572,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":204,"threads_started":1,"wal-append.queue_time_us":292}
I20240501 14:07:21.545382 5302 maintenance_manager.cc:419] P 1c3fdc5272484ac09ef3e5e36a08f27c: Scheduling UndoDeltaBlockGCOp(23ca2bf1e59f44fcac8a48aff8d056b1): 4834 bytes on disk
I20240501 14:07:21.545847 5193 maintenance_manager.cc:643] P 1c3fdc5272484ac09ef3e5e36a08f27c: UndoDeltaBlockGCOp(23ca2bf1e59f44fcac8a48aff8d056b1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4}
I20240501 14:07:21.554109 5041 maintenance_manager.cc:419] P 04088ee3bea440a7bd5ff772152b9410: Scheduling FlushMRSOp(71bdb382b08341218c8bf2459fe65fc0): perf score=0.033365
I20240501 14:07:21.563585 5432 maintenance_manager.cc:419] P be56773105e744919601127a94e98adf: Scheduling FlushMRSOp(71bdb382b08341218c8bf2459fe65fc0): perf score=0.033370
I20240501 14:07:21.580916 18904 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:07:21.581094 18904 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:07:21.581871 18904 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:07:21.582773 5255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" is_pre_election: true
I20240501 14:07:21.583006 5255 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 2.
W20240501 14:07:21.585570 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:21.585748 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:07:21.585840 5321 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:07:21.586076 18904 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:07:21.591204 4932 maintenance_manager.cc:643] P 04088ee3bea440a7bd5ff772152b9410: FlushMRSOp(71bdb382b08341218c8bf2459fe65fc0) complete. Timing: real 0.037s user 0.026s sys 0.005s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":2489,"dirs.run_cpu_time_us":225,"dirs.run_wall_time_us":1116,"drs_written":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4,"lbm_write_time_us":639,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":2588,"threads_started":2,"wal-append.queue_time_us":303}
I20240501 14:07:21.592185 5041 maintenance_manager.cc:419] P 04088ee3bea440a7bd5ff772152b9410: Scheduling FlushMRSOp(63650547ab444f45bb377bcb49f258ad): perf score=0.033372
I20240501 14:07:21.618029 5323 maintenance_manager.cc:643] P be56773105e744919601127a94e98adf: FlushMRSOp(71bdb382b08341218c8bf2459fe65fc0) complete. Timing: real 0.054s user 0.039s sys 0.013s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":309,"dirs.run_cpu_time_us":223,"dirs.run_wall_time_us":1435,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":729,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":452,"threads_started":2,"wal-append.queue_time_us":363}
I20240501 14:07:21.619648 5432 maintenance_manager.cc:419] P be56773105e744919601127a94e98adf: Scheduling UndoDeltaBlockGCOp(71bdb382b08341218c8bf2459fe65fc0): 4581 bytes on disk
I20240501 14:07:21.620290 5323 maintenance_manager.cc:643] P be56773105e744919601127a94e98adf: UndoDeltaBlockGCOp(71bdb382b08341218c8bf2459fe65fc0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4}
W20240501 14:07:21.625219 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
I20240501 14:07:21.649191 4932 maintenance_manager.cc:643] P 04088ee3bea440a7bd5ff772152b9410: FlushMRSOp(63650547ab444f45bb377bcb49f258ad) complete. Timing: real 0.057s user 0.035s sys 0.020s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":2791,"dirs.run_cpu_time_us":202,"dirs.run_wall_time_us":1187,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":765,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":13304,"threads_started":1,"wal-append.queue_time_us":13403}
I20240501 14:07:21.650435 5041 maintenance_manager.cc:419] P 04088ee3bea440a7bd5ff772152b9410: Scheduling UndoDeltaBlockGCOp(63650547ab444f45bb377bcb49f258ad): 4921 bytes on disk
I20240501 14:07:21.650949 4932 maintenance_manager.cc:643] P 04088ee3bea440a7bd5ff772152b9410: UndoDeltaBlockGCOp(63650547ab444f45bb377bcb49f258ad) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
I20240501 14:07:21.652002 5041 maintenance_manager.cc:419] P 04088ee3bea440a7bd5ff772152b9410: Scheduling UndoDeltaBlockGCOp(71bdb382b08341218c8bf2459fe65fc0): 4581 bytes on disk
I20240501 14:07:21.652462 4932 maintenance_manager.cc:643] P 04088ee3bea440a7bd5ff772152b9410: UndoDeltaBlockGCOp(71bdb382b08341218c8bf2459fe65fc0) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
W20240501 14:07:21.818449 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:07:21.879825 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:22.009016 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:07:22.049908 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:07:22.109414 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:07:22.121477 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:07:22.137923 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:07:22.149562 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:07:22.166694 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:22.291402 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:07:22.374375 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 14:07:22.405704 5041 maintenance_manager.cc:419] P 04088ee3bea440a7bd5ff772152b9410: Scheduling FlushMRSOp(99d958617b7e4ef385b94f4c73f4dd84): perf score=0.033337
I20240501 14:07:22.435252 4932 maintenance_manager.cc:643] P 04088ee3bea440a7bd5ff772152b9410: FlushMRSOp(99d958617b7e4ef385b94f4c73f4dd84) complete. Timing: real 0.029s user 0.023s sys 0.004s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":318,"dirs.run_cpu_time_us":211,"dirs.run_wall_time_us":1391,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":543,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":391,"threads_started":2,"wal-append.queue_time_us":292}
I20240501 14:07:22.436249 5041 maintenance_manager.cc:419] P 04088ee3bea440a7bd5ff772152b9410: Scheduling UndoDeltaBlockGCOp(99d958617b7e4ef385b94f4c73f4dd84): 4834 bytes on disk
I20240501 14:07:22.436729 4932 maintenance_manager.cc:643] P 04088ee3bea440a7bd5ff772152b9410: UndoDeltaBlockGCOp(99d958617b7e4ef385b94f4c73f4dd84) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
W20240501 14:07:22.469522 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:07:22.522269 5172 maintenance_manager.cc:419] P e9c60449cf3c46f8aca4ab6c535302a4: Scheduling FlushMRSOp(3d6859d53fd747a5a5e94722626fa522): perf score=0.033370
I20240501 14:07:22.580190 5063 maintenance_manager.cc:643] P e9c60449cf3c46f8aca4ab6c535302a4: FlushMRSOp(3d6859d53fd747a5a5e94722626fa522) complete. Timing: real 0.058s user 0.035s sys 0.019s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":311,"dirs.run_cpu_time_us":214,"dirs.run_wall_time_us":1522,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":618,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":14181,"threads_started":2,"wal-append.queue_time_us":14056}
I20240501 14:07:22.581205 5172 maintenance_manager.cc:419] P e9c60449cf3c46f8aca4ab6c535302a4: Scheduling FlushMRSOp(9d32e0707d244f2c9bbcb57f62b09f31): perf score=0.033383
I20240501 14:07:22.613540 5063 maintenance_manager.cc:643] P e9c60449cf3c46f8aca4ab6c535302a4: FlushMRSOp(9d32e0707d244f2c9bbcb57f62b09f31) complete. Timing: real 0.032s user 0.027s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":49,"dirs.run_cpu_time_us":206,"dirs.run_wall_time_us":1451,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":636,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":187,"threads_started":1,"wal-append.queue_time_us":281}
I20240501 14:07:22.614809 5172 maintenance_manager.cc:419] P e9c60449cf3c46f8aca4ab6c535302a4: Scheduling UndoDeltaBlockGCOp(3d6859d53fd747a5a5e94722626fa522): 4921 bytes on disk
I20240501 14:07:22.615384 5063 maintenance_manager.cc:643] P e9c60449cf3c46f8aca4ab6c535302a4: UndoDeltaBlockGCOp(3d6859d53fd747a5a5e94722626fa522) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
I20240501 14:07:22.616319 5172 maintenance_manager.cc:419] P e9c60449cf3c46f8aca4ab6c535302a4: Scheduling UndoDeltaBlockGCOp(9d32e0707d244f2c9bbcb57f62b09f31): 4581 bytes on disk
I20240501 14:07:22.616816 5063 maintenance_manager.cc:643] P e9c60449cf3c46f8aca4ab6c535302a4: UndoDeltaBlockGCOp(9d32e0707d244f2c9bbcb57f62b09f31) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4}
I20240501 14:07:22.624632 5432 maintenance_manager.cc:419] P be56773105e744919601127a94e98adf: Scheduling FlushMRSOp(9d32e0707d244f2c9bbcb57f62b09f31): perf score=0.033397
I20240501 14:07:22.658847 5323 maintenance_manager.cc:643] P be56773105e744919601127a94e98adf: FlushMRSOp(9d32e0707d244f2c9bbcb57f62b09f31) complete. Timing: real 0.034s user 0.027s sys 0.005s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":261,"dirs.run_cpu_time_us":173,"dirs.run_wall_time_us":1299,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":660,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":383,"threads_started":2,"wal-append.queue_time_us":302}
I20240501 14:07:22.659868 5432 maintenance_manager.cc:419] P be56773105e744919601127a94e98adf: Scheduling FlushMRSOp(99d958617b7e4ef385b94f4c73f4dd84): perf score=0.033403
I20240501 14:07:22.691751 5323 maintenance_manager.cc:643] P be56773105e744919601127a94e98adf: FlushMRSOp(99d958617b7e4ef385b94f4c73f4dd84) complete. Timing: real 0.032s user 0.021s sys 0.008s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":56,"dirs.run_cpu_time_us":186,"dirs.run_wall_time_us":1660,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":569,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":199,"threads_started":1,"wal-append.queue_time_us":308}
I20240501 14:07:22.692812 5432 maintenance_manager.cc:419] P be56773105e744919601127a94e98adf: Scheduling UndoDeltaBlockGCOp(99d958617b7e4ef385b94f4c73f4dd84): 4834 bytes on disk
I20240501 14:07:22.693312 5323 maintenance_manager.cc:643] P be56773105e744919601127a94e98adf: UndoDeltaBlockGCOp(99d958617b7e4ef385b94f4c73f4dd84) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20240501 14:07:22.694198 5432 maintenance_manager.cc:419] P be56773105e744919601127a94e98adf: Scheduling UndoDeltaBlockGCOp(9d32e0707d244f2c9bbcb57f62b09f31): 4581 bytes on disk
I20240501 14:07:22.694717 5323 maintenance_manager.cc:643] P be56773105e744919601127a94e98adf: UndoDeltaBlockGCOp(9d32e0707d244f2c9bbcb57f62b09f31) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4}
W20240501 14:07:22.773031 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:22.784781 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:07:22.847757 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:23.096961 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:07:23.160463 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:23.222048 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:07:23.225720 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:07:23.289033 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:07:23.298171 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:23.342185 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:07:23.388936 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:07:23.393491 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 14:07:23.440652 5041 maintenance_manager.cc:419] P 04088ee3bea440a7bd5ff772152b9410: Scheduling FlushMRSOp(ec1e81011c6a46bc8a39aecf3a5e71e4): perf score=0.033367
W20240501 14:07:23.442497 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 14:07:23.448156 5432 maintenance_manager.cc:419] P be56773105e744919601127a94e98adf: Scheduling FlushMRSOp(420642af520d45ffbc7f2d35f93e75a7): perf score=0.033369
I20240501 14:07:23.470430 4932 maintenance_manager.cc:643] P 04088ee3bea440a7bd5ff772152b9410: FlushMRSOp(ec1e81011c6a46bc8a39aecf3a5e71e4) complete. Timing: real 0.030s user 0.009s sys 0.020s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":319,"dirs.run_cpu_time_us":179,"dirs.run_wall_time_us":1431,"drs_written":1,"lbm_read_time_us":48,"lbm_reads_lt_1ms":4,"lbm_write_time_us":546,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":382,"threads_started":2,"wal-append.queue_time_us":284}
I20240501 14:07:23.471613 5041 maintenance_manager.cc:419] P 04088ee3bea440a7bd5ff772152b9410: Scheduling UndoDeltaBlockGCOp(ec1e81011c6a46bc8a39aecf3a5e71e4): 4834 bytes on disk
I20240501 14:07:23.472205 4932 maintenance_manager.cc:643] P 04088ee3bea440a7bd5ff772152b9410: UndoDeltaBlockGCOp(ec1e81011c6a46bc8a39aecf3a5e71e4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4}
I20240501 14:07:23.500684 5323 maintenance_manager.cc:643] P be56773105e744919601127a94e98adf: FlushMRSOp(420642af520d45ffbc7f2d35f93e75a7) complete. Timing: real 0.052s user 0.035s sys 0.014s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":281,"dirs.run_cpu_time_us":214,"dirs.run_wall_time_us":1646,"drs_written":1,"lbm_read_time_us":57,"lbm_reads_lt_1ms":4,"lbm_write_time_us":579,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":766,"threads_started":2,"wal-append.queue_time_us":306}
I20240501 14:07:23.501972 5432 maintenance_manager.cc:419] P be56773105e744919601127a94e98adf: Scheduling FlushMRSOp(61a274b24f53458f88c2cde4f8dbe4f4): perf score=0.033380
I20240501 14:07:23.545214 5323 maintenance_manager.cc:643] P be56773105e744919601127a94e98adf: FlushMRSOp(61a274b24f53458f88c2cde4f8dbe4f4) complete. Timing: real 0.043s user 0.022s sys 0.017s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":56,"dirs.run_cpu_time_us":210,"dirs.run_wall_time_us":1645,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":767,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":1049,"threads_started":1,"wal-append.queue_time_us":1149}
I20240501 14:07:23.546594 5432 maintenance_manager.cc:419] P be56773105e744919601127a94e98adf: Scheduling UndoDeltaBlockGCOp(61a274b24f53458f88c2cde4f8dbe4f4): 4921 bytes on disk
I20240501 14:07:23.547242 5323 maintenance_manager.cc:643] P be56773105e744919601127a94e98adf: UndoDeltaBlockGCOp(61a274b24f53458f88c2cde4f8dbe4f4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4}
I20240501 14:07:23.548243 5432 maintenance_manager.cc:419] P be56773105e744919601127a94e98adf: Scheduling UndoDeltaBlockGCOp(420642af520d45ffbc7f2d35f93e75a7): 4579 bytes on disk
I20240501 14:07:23.548645 5323 maintenance_manager.cc:643] P be56773105e744919601127a94e98adf: UndoDeltaBlockGCOp(420642af520d45ffbc7f2d35f93e75a7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
I20240501 14:07:23.552845 5302 maintenance_manager.cc:419] P 1c3fdc5272484ac09ef3e5e36a08f27c: Scheduling FlushMRSOp(420642af520d45ffbc7f2d35f93e75a7): perf score=0.033395
I20240501 14:07:23.606732 5193 maintenance_manager.cc:643] P 1c3fdc5272484ac09ef3e5e36a08f27c: FlushMRSOp(420642af520d45ffbc7f2d35f93e75a7) complete. Timing: real 0.054s user 0.035s sys 0.017s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":268,"dirs.run_cpu_time_us":204,"dirs.run_wall_time_us":1289,"drs_written":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4,"lbm_write_time_us":659,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":394,"threads_started":2,"wal-append.queue_time_us":329}
I20240501 14:07:23.607710 5302 maintenance_manager.cc:419] P 1c3fdc5272484ac09ef3e5e36a08f27c: Scheduling FlushMRSOp(ec1e81011c6a46bc8a39aecf3a5e71e4): perf score=0.033407
W20240501 14:07:23.641551 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240501 14:07:23.645524 5193 maintenance_manager.cc:643] P 1c3fdc5272484ac09ef3e5e36a08f27c: FlushMRSOp(ec1e81011c6a46bc8a39aecf3a5e71e4) complete. Timing: real 0.038s user 0.020s sys 0.016s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":46,"dirs.run_cpu_time_us":185,"dirs.run_wall_time_us":1164,"drs_written":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4,"lbm_write_time_us":725,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":197,"threads_started":1,"wal-append.queue_time_us":281}
I20240501 14:07:23.646566 5302 maintenance_manager.cc:419] P 1c3fdc5272484ac09ef3e5e36a08f27c: Scheduling UndoDeltaBlockGCOp(ec1e81011c6a46bc8a39aecf3a5e71e4): 4834 bytes on disk
I20240501 14:07:23.647099 5193 maintenance_manager.cc:643] P 1c3fdc5272484ac09ef3e5e36a08f27c: UndoDeltaBlockGCOp(ec1e81011c6a46bc8a39aecf3a5e71e4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
I20240501 14:07:23.647908 5302 maintenance_manager.cc:419] P 1c3fdc5272484ac09ef3e5e36a08f27c: Scheduling UndoDeltaBlockGCOp(420642af520d45ffbc7f2d35f93e75a7): 4579 bytes on disk
I20240501 14:07:23.648445 5193 maintenance_manager.cc:643] P 1c3fdc5272484ac09ef3e5e36a08f27c: UndoDeltaBlockGCOp(420642af520d45ffbc7f2d35f93e75a7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4}
W20240501 14:07:23.698664 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:23.874328 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:23.971711 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:07:24.040400 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:07:24.217525 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:24.273882 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 14:07:24.303794 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 74 similar messages]
W20240501 14:07:24.304596 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:07:24.332545 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:07:24.346987 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:07:24.376330 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:24.600139 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:07:24.614662 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:24.628659 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:07:24.651932 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:07:24.706933 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 14:07:24.748098 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:07:24.894814 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:24.976265 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:07:25.033800 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:07:25.188298 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:25.190749 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240501 14:07:25.316560 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:07:25.460340 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:25.581064 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:25.708200 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:07:25.748512 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:07:25.755442 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:07:25.769573 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:07:25.828238 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:07:25.845741 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:07:25.863605 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:07:25.889071 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:25.907408 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:07:26.040076 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:26.082461 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:07:26.300946 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:26.427093 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:26.490177 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:07:26.557202 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:07:26.810232 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:07:26.814292 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:07:26.825984 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:07:26.832233 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:26.861236 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:07:26.874164 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:07:26.899672 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:27.021571 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:07:27.058056 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:07:27.226599 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:27.351158 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:27.505743 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:07:27.703133 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:07:27.704411 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:27.799893 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:07:27.883175 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:28.026052 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:07:28.041326 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:07:28.157310 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:28.217371 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:07:28.345336 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:07:28.353389 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:07:28.373481 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:28.408784 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:07:28.425558 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:07:28.486765 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:07:28.552206 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:28.623906 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:07:28.792330 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:07:28.841440 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:29.132257 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:29.152390 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:07:29.185763 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:07:29.256739 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:07:29.284370 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:07:29.322046 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240501 14:07:29.322850 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:29.338793 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 14:07:29.369683 19722 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:07:29.369916 19722 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:07:29.370781 19722 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers be56773105e744919601127a94e98adf (127.29.124.132:35459), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:07:29.371083 5384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf" is_pre_election: true
I20240501 14:07:29.371301 5384 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 2.
W20240501 14:07:29.384258 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:29.384490 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:07:29.384579 5191 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:07:29.402454 19722 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:07:29.431385 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 74 similar messages]
W20240501 14:07:29.454599 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:07:29.476109 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:07:29.558782 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:29.595538 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:07:29.729348 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:29.845610 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 14:07:29.988868 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:07:30.073890 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:30.132688 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:30.265470 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:07:30.351039 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 14:07:30.359613 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:07:30.365572 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:07:30.451391 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:30.479948 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:07:30.548728 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:30.841199 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:07:30.850725 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:07:30.914006 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:07:30.968747 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:07:30.975121 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:07:31.006425 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:07:31.064289 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:31.111198 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:07:31.160082 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:31.227969 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:07:31.483937 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:07:31.540467 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:31.592129 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:31.650851 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:07:31.789657 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:07:31.867164 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:07:31.885890 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:07:31.981848 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:32.010198 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:07:32.063704 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:07:32.158838 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:32.179034 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:07:32.395507 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:32.402006 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240501 14:07:32.667279 7334 maintenance_manager.cc:419] P 6dac90f01e744f9c8701a404440a4c77: Scheduling FlushMRSOp(de6c7753c8564107b858876fcfa5924d): perf score=0.033399
W20240501 14:07:32.709650 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
I20240501 14:07:32.739802 7224 maintenance_manager.cc:643] P 6dac90f01e744f9c8701a404440a4c77: FlushMRSOp(de6c7753c8564107b858876fcfa5924d) complete. Timing: real 0.072s user 0.068s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":397,"dirs.queue_time_us":377,"dirs.run_cpu_time_us":175,"dirs.run_wall_time_us":1183,"drs_written":1,"lbm_read_time_us":121,"lbm_reads_lt_1ms":4,"lbm_write_time_us":892,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":740,"threads_started":3,"wal-append.queue_time_us":357}
I20240501 14:07:32.741816 7334 maintenance_manager.cc:419] P 6dac90f01e744f9c8701a404440a4c77: Scheduling UndoDeltaBlockGCOp(de6c7753c8564107b858876fcfa5924d): 4581 bytes on disk
I20240501 14:07:32.742736 7224 maintenance_manager.cc:643] P 6dac90f01e744f9c8701a404440a4c77: UndoDeltaBlockGCOp(de6c7753c8564107b858876fcfa5924d) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4}
W20240501 14:07:32.784667 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:07:32.825323 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:32.852536 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:07:32.872337 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 14:07:32.994396 7334 maintenance_manager.cc:419] P 6dac90f01e744f9c8701a404440a4c77: Scheduling FlushMRSOp(9d32e0707d244f2c9bbcb57f62b09f31): perf score=0.033402
I20240501 14:07:33.031472 7224 maintenance_manager.cc:643] P 6dac90f01e744f9c8701a404440a4c77: FlushMRSOp(9d32e0707d244f2c9bbcb57f62b09f31) complete. Timing: real 0.037s user 0.026s sys 0.008s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":57,"dirs.run_cpu_time_us":230,"dirs.run_wall_time_us":1468,"drs_written":1,"lbm_read_time_us":91,"lbm_reads_lt_1ms":4,"lbm_write_time_us":710,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":247,"threads_started":1,"wal-append.queue_time_us":361}
I20240501 14:07:33.033000 7334 maintenance_manager.cc:419] P 6dac90f01e744f9c8701a404440a4c77: Scheduling UndoDeltaBlockGCOp(9d32e0707d244f2c9bbcb57f62b09f31): 4581 bytes on disk
I20240501 14:07:33.033625 7224 maintenance_manager.cc:643] P 6dac90f01e744f9c8701a404440a4c77: UndoDeltaBlockGCOp(9d32e0707d244f2c9bbcb57f62b09f31) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
W20240501 14:07:33.197021 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:07:33.284376 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
I20240501 14:07:33.285315 7334 maintenance_manager.cc:419] P 6dac90f01e744f9c8701a404440a4c77: Scheduling FlushMRSOp(f9cb784f60774ea7b6e70ccfebf867a8): perf score=0.033344
W20240501 14:07:33.307011 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 14:07:33.317824 7224 maintenance_manager.cc:643] P 6dac90f01e744f9c8701a404440a4c77: FlushMRSOp(f9cb784f60774ea7b6e70ccfebf867a8) complete. Timing: real 0.032s user 0.021s sys 0.011s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":50,"dirs.run_cpu_time_us":200,"dirs.run_wall_time_us":1120,"drs_written":1,"lbm_read_time_us":64,"lbm_reads_lt_1ms":4,"lbm_write_time_us":649,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":178,"threads_started":1,"wal-append.queue_time_us":274}
I20240501 14:07:33.318856 7334 maintenance_manager.cc:419] P 6dac90f01e744f9c8701a404440a4c77: Scheduling UndoDeltaBlockGCOp(f9cb784f60774ea7b6e70ccfebf867a8): 4921 bytes on disk
I20240501 14:07:33.319365 7224 maintenance_manager.cc:643] P 6dac90f01e744f9c8701a404440a4c77: UndoDeltaBlockGCOp(f9cb784f60774ea7b6e70ccfebf867a8) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4}
W20240501 14:07:33.327201 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:07:33.379662 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:07:33.428089 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:07:33.505280 7071 maintenance_manager.cc:419] P 19a6d2d8a08142d5bffcde3a37234692: Scheduling FlushMRSOp(8cc5eec4600640f7a293877c825f8d96): perf score=0.033369
W20240501 14:07:33.540005 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:07:33.592787 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 14:07:33.600508 6962 maintenance_manager.cc:643] P 19a6d2d8a08142d5bffcde3a37234692: FlushMRSOp(8cc5eec4600640f7a293877c825f8d96) complete. Timing: real 0.094s user 0.045s sys 0.029s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":445,"dirs.queue_time_us":605,"dirs.run_cpu_time_us":214,"dirs.run_wall_time_us":833,"drs_written":1,"lbm_read_time_us":108,"lbm_reads_lt_1ms":4,"lbm_write_time_us":849,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":961,"threads_started":3,"wal-append.queue_time_us":375}
I20240501 14:07:33.603037 7071 maintenance_manager.cc:419] P 19a6d2d8a08142d5bffcde3a37234692: Scheduling UndoDeltaBlockGCOp(8cc5eec4600640f7a293877c825f8d96): 4834 bytes on disk
I20240501 14:07:33.603967 6962 maintenance_manager.cc:643] P 19a6d2d8a08142d5bffcde3a37234692: UndoDeltaBlockGCOp(8cc5eec4600640f7a293877c825f8d96) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
I20240501 14:07:33.605293 7071 maintenance_manager.cc:419] P 19a6d2d8a08142d5bffcde3a37234692: Scheduling FlushMRSOp(1a7bbfc15a76450bbc9197037e856e08): perf score=0.033389
W20240501 14:07:33.642482 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240501 14:07:33.658502 6962 maintenance_manager.cc:643] P 19a6d2d8a08142d5bffcde3a37234692: FlushMRSOp(1a7bbfc15a76450bbc9197037e856e08) complete. Timing: real 0.053s user 0.045s sys 0.003s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":43,"dirs.run_cpu_time_us":166,"dirs.run_wall_time_us":1031,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":784,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":275,"threads_started":1,"wal-append.queue_time_us":392}
I20240501 14:07:33.659518 7071 maintenance_manager.cc:419] P 19a6d2d8a08142d5bffcde3a37234692: Scheduling UndoDeltaBlockGCOp(1a7bbfc15a76450bbc9197037e856e08): 4921 bytes on disk
I20240501 14:07:33.660034 6962 maintenance_manager.cc:643] P 19a6d2d8a08142d5bffcde3a37234692: UndoDeltaBlockGCOp(1a7bbfc15a76450bbc9197037e856e08) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
W20240501 14:07:33.773016 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:07:33.845227 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:33.892483 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:07:33.983474 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:07:34.021764 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:07:34.073026 7334 maintenance_manager.cc:419] P 6dac90f01e744f9c8701a404440a4c77: Scheduling FlushMRSOp(9afe3b20a43d4b5bb800af39f945948d): perf score=0.033364
I20240501 14:07:34.124648 7224 maintenance_manager.cc:643] P 6dac90f01e744f9c8701a404440a4c77: FlushMRSOp(9afe3b20a43d4b5bb800af39f945948d) complete. Timing: real 0.051s user 0.033s sys 0.013s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":691,"dirs.run_cpu_time_us":168,"dirs.run_wall_time_us":1497,"drs_written":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4,"lbm_write_time_us":613,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":1647,"threads_started":2,"wal-append.queue_time_us":744}
I20240501 14:07:34.126057 7334 maintenance_manager.cc:419] P 6dac90f01e744f9c8701a404440a4c77: Scheduling UndoDeltaBlockGCOp(9afe3b20a43d4b5bb800af39f945948d): 4921 bytes on disk
I20240501 14:07:34.127045 7224 maintenance_manager.cc:643] P 6dac90f01e744f9c8701a404440a4c77: UndoDeltaBlockGCOp(9afe3b20a43d4b5bb800af39f945948d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
I20240501 14:07:34.162873 7071 maintenance_manager.cc:419] P 19a6d2d8a08142d5bffcde3a37234692: Scheduling FlushMRSOp(f4d2b8c005b64bbb8e6621da068d15e5): perf score=0.033401
I20240501 14:07:34.203327 6962 maintenance_manager.cc:643] P 19a6d2d8a08142d5bffcde3a37234692: FlushMRSOp(f4d2b8c005b64bbb8e6621da068d15e5) complete. Timing: real 0.040s user 0.030s sys 0.008s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":338,"dirs.run_cpu_time_us":199,"dirs.run_wall_time_us":1198,"drs_written":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4,"lbm_write_time_us":558,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":420,"threads_started":2,"wal-append.queue_time_us":354}
I20240501 14:07:34.204317 7071 maintenance_manager.cc:419] P 19a6d2d8a08142d5bffcde3a37234692: Scheduling UndoDeltaBlockGCOp(f4d2b8c005b64bbb8e6621da068d15e5): 4581 bytes on disk
I20240501 14:07:34.204918 6962 maintenance_manager.cc:643] P 19a6d2d8a08142d5bffcde3a37234692: UndoDeltaBlockGCOp(f4d2b8c005b64bbb8e6621da068d15e5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4}
W20240501 14:07:34.313853 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:07:34.366441 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:34.372359 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:07:34.405295 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:07:34.416481 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 14:07:34.417171 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:07:34.436146 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:34.552304 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240501 14:07:34.553107 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:07:34.589200 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:07:34.691175 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:07:34.759527 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
I20240501 14:07:34.958287 7071 maintenance_manager.cc:419] P 19a6d2d8a08142d5bffcde3a37234692: Scheduling FlushMRSOp(23ca2bf1e59f44fcac8a48aff8d056b1): perf score=0.033391
W20240501 14:07:34.967012 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:07:34.992038 6962 maintenance_manager.cc:643] P 19a6d2d8a08142d5bffcde3a37234692: FlushMRSOp(23ca2bf1e59f44fcac8a48aff8d056b1) complete. Timing: real 0.033s user 0.015s sys 0.016s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":272,"dirs.run_cpu_time_us":191,"dirs.run_wall_time_us":1117,"drs_written":1,"lbm_read_time_us":46,"lbm_reads_lt_1ms":4,"lbm_write_time_us":611,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"spinlock_wait_cycles":2816,"thread_start_us":355,"threads_started":2,"wal-append.queue_time_us":258}
I20240501 14:07:34.993052 7071 maintenance_manager.cc:419] P 19a6d2d8a08142d5bffcde3a37234692: Scheduling UndoDeltaBlockGCOp(23ca2bf1e59f44fcac8a48aff8d056b1): 4834 bytes on disk
I20240501 14:07:34.993561 6962 maintenance_manager.cc:643] P 19a6d2d8a08142d5bffcde3a37234692: UndoDeltaBlockGCOp(23ca2bf1e59f44fcac8a48aff8d056b1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4}
W20240501 14:07:35.002810 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 63 similar messages]
W20240501 14:07:35.040024 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:07:35.049407 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:07:35.283643 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:35.338678 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:07:35.465341 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 14:07:35.496206 7071 maintenance_manager.cc:419] P 19a6d2d8a08142d5bffcde3a37234692: Scheduling FlushMRSOp(61a274b24f53458f88c2cde4f8dbe4f4): perf score=0.033368
I20240501 14:07:35.531881 6962 maintenance_manager.cc:643] P 19a6d2d8a08142d5bffcde3a37234692: FlushMRSOp(61a274b24f53458f88c2cde4f8dbe4f4) complete. Timing: real 0.035s user 0.015s sys 0.018s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":359,"dirs.run_cpu_time_us":146,"dirs.run_wall_time_us":1353,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":822,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":470,"threads_started":2,"wal-append.queue_time_us":310}
I20240501 14:07:35.533118 7071 maintenance_manager.cc:419] P 19a6d2d8a08142d5bffcde3a37234692: Scheduling UndoDeltaBlockGCOp(61a274b24f53458f88c2cde4f8dbe4f4): 4921 bytes on disk
I20240501 14:07:35.533720 6962 maintenance_manager.cc:643] P 19a6d2d8a08142d5bffcde3a37234692: UndoDeltaBlockGCOp(61a274b24f53458f88c2cde4f8dbe4f4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
W20240501 14:07:35.537572 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:35.644771 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:07:35.653913 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240501 14:07:35.699059 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:07:35.719926 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
I20240501 14:07:35.785135 7071 maintenance_manager.cc:419] P 19a6d2d8a08142d5bffcde3a37234692: Scheduling FlushMRSOp(e7d5f02ccba34254a4d20951878b1214): perf score=0.033385
W20240501 14:07:35.791749 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 14:07:35.825918 6962 maintenance_manager.cc:643] P 19a6d2d8a08142d5bffcde3a37234692: FlushMRSOp(e7d5f02ccba34254a4d20951878b1214) complete. Timing: real 0.041s user 0.025s sys 0.015s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":38,"dirs.run_cpu_time_us":173,"dirs.run_wall_time_us":1288,"drs_written":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4,"lbm_write_time_us":638,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":206,"threads_started":1,"wal-append.queue_time_us":299}
I20240501 14:07:35.826987 7071 maintenance_manager.cc:419] P 19a6d2d8a08142d5bffcde3a37234692: Scheduling UndoDeltaBlockGCOp(e7d5f02ccba34254a4d20951878b1214): 4834 bytes on disk
I20240501 14:07:35.827476 6962 maintenance_manager.cc:643] P 19a6d2d8a08142d5bffcde3a37234692: UndoDeltaBlockGCOp(e7d5f02ccba34254a4d20951878b1214) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
W20240501 14:07:35.872377 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:07:36.001165 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:07:36.020365 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:36.038802 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:07:36.199098 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:36.225157 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:07:36.259925 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:07:36.272367 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 14:07:36.329892 7071 maintenance_manager.cc:419] P 19a6d2d8a08142d5bffcde3a37234692: Scheduling FlushMRSOp(404e2851055541f0be1bb0971a7cc72a): perf score=0.033373
I20240501 14:07:36.361182 6962 maintenance_manager.cc:643] P 19a6d2d8a08142d5bffcde3a37234692: FlushMRSOp(404e2851055541f0be1bb0971a7cc72a) complete. Timing: real 0.031s user 0.016s sys 0.012s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":306,"dirs.run_cpu_time_us":171,"dirs.run_wall_time_us":1294,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":652,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":358,"threads_started":2,"wal-append.queue_time_us":237}
I20240501 14:07:36.362284 7071 maintenance_manager.cc:419] P 19a6d2d8a08142d5bffcde3a37234692: Scheduling UndoDeltaBlockGCOp(404e2851055541f0be1bb0971a7cc72a): 4834 bytes on disk
I20240501 14:07:36.362797 6962 maintenance_manager.cc:643] P 19a6d2d8a08142d5bffcde3a37234692: UndoDeltaBlockGCOp(404e2851055541f0be1bb0971a7cc72a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4}
I20240501 14:07:36.385366 7334 maintenance_manager.cc:419] P 6dac90f01e744f9c8701a404440a4c77: Scheduling FlushMRSOp(e7d5f02ccba34254a4d20951878b1214): perf score=0.033382
I20240501 14:07:36.429567 7224 maintenance_manager.cc:643] P 6dac90f01e744f9c8701a404440a4c77: FlushMRSOp(e7d5f02ccba34254a4d20951878b1214) complete. Timing: real 0.044s user 0.031s sys 0.012s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":5413,"dirs.run_cpu_time_us":151,"dirs.run_wall_time_us":1266,"drs_written":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4,"lbm_write_time_us":600,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":5497,"threads_started":2,"wal-append.queue_time_us":267}
I20240501 14:07:36.430634 7334 maintenance_manager.cc:419] P 6dac90f01e744f9c8701a404440a4c77: Scheduling FlushMRSOp(61a274b24f53458f88c2cde4f8dbe4f4): perf score=0.033395
I20240501 14:07:36.465077 7224 maintenance_manager.cc:643] P 6dac90f01e744f9c8701a404440a4c77: FlushMRSOp(61a274b24f53458f88c2cde4f8dbe4f4) complete. Timing: real 0.034s user 0.020s sys 0.012s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":57,"dirs.run_cpu_time_us":180,"dirs.run_wall_time_us":1292,"drs_written":1,"lbm_read_time_us":47,"lbm_reads_lt_1ms":4,"lbm_write_time_us":537,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":191,"threads_started":1,"wal-append.queue_time_us":283}
I20240501 14:07:36.466076 7334 maintenance_manager.cc:419] P 6dac90f01e744f9c8701a404440a4c77: Scheduling FlushMRSOp(2c39b53867e047faaf24cc6f87a8100c): perf score=0.033390
W20240501 14:07:36.500898 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:07:36.504362 7224 maintenance_manager.cc:643] P 6dac90f01e744f9c8701a404440a4c77: FlushMRSOp(2c39b53867e047faaf24cc6f87a8100c) complete. Timing: real 0.038s user 0.035s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":52,"dirs.run_cpu_time_us":314,"dirs.run_wall_time_us":1254,"drs_written":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4,"lbm_write_time_us":553,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":213,"threads_started":1,"wal-append.queue_time_us":300}
I20240501 14:07:36.505378 7334 maintenance_manager.cc:419] P 6dac90f01e744f9c8701a404440a4c77: Scheduling UndoDeltaBlockGCOp(61a274b24f53458f88c2cde4f8dbe4f4): 4921 bytes on disk
I20240501 14:07:36.505837 7224 maintenance_manager.cc:643] P 6dac90f01e744f9c8701a404440a4c77: UndoDeltaBlockGCOp(61a274b24f53458f88c2cde4f8dbe4f4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
I20240501 14:07:36.506609 7334 maintenance_manager.cc:419] P 6dac90f01e744f9c8701a404440a4c77: Scheduling UndoDeltaBlockGCOp(e7d5f02ccba34254a4d20951878b1214): 4834 bytes on disk
I20240501 14:07:36.506987 7224 maintenance_manager.cc:643] P 6dac90f01e744f9c8701a404440a4c77: UndoDeltaBlockGCOp(e7d5f02ccba34254a4d20951878b1214) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":55,"lbm_reads_lt_1ms":4}
I20240501 14:07:36.507679 7334 maintenance_manager.cc:419] P 6dac90f01e744f9c8701a404440a4c77: Scheduling UndoDeltaBlockGCOp(2c39b53867e047faaf24cc6f87a8100c): 4581 bytes on disk
I20240501 14:07:36.508114 7224 maintenance_manager.cc:643] P 6dac90f01e744f9c8701a404440a4c77: UndoDeltaBlockGCOp(2c39b53867e047faaf24cc6f87a8100c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4}
W20240501 14:07:36.637521 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:07:36.717089 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:07:36.726426 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:36.791429 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 14:07:36.865505 7071 maintenance_manager.cc:419] P 19a6d2d8a08142d5bffcde3a37234692: Scheduling FlushMRSOp(43b46fcf827c4275a47291afb107c290): perf score=0.033394
W20240501 14:07:36.882926 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 14:07:36.906526 6962 maintenance_manager.cc:643] P 19a6d2d8a08142d5bffcde3a37234692: FlushMRSOp(43b46fcf827c4275a47291afb107c290) complete. Timing: real 0.040s user 0.026s sys 0.012s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":380,"dirs.run_cpu_time_us":214,"dirs.run_wall_time_us":1344,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":732,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":459,"threads_started":2,"wal-append.queue_time_us":261}
I20240501 14:07:36.907516 7071 maintenance_manager.cc:419] P 19a6d2d8a08142d5bffcde3a37234692: Scheduling UndoDeltaBlockGCOp(43b46fcf827c4275a47291afb107c290): 4833 bytes on disk
I20240501 14:07:36.907946 6962 maintenance_manager.cc:643] P 19a6d2d8a08142d5bffcde3a37234692: UndoDeltaBlockGCOp(43b46fcf827c4275a47291afb107c290) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":58,"lbm_reads_lt_1ms":4}
W20240501 14:07:36.953522 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:07:36.964181 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:07:36.988041 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:37.177503 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:07:37.182744 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240501 14:07:37.201736 20805 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:07:37.201895 20805 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:07:37.202699 20805 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:07:37.203138 5254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" is_pre_election: true
I20240501 14:07:37.203362 5254 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 2.
W20240501 14:07:37.204463 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:37.204641 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:07:37.204741 5321 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:07:37.204960 20805 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:07:37.328689 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:37.376634 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:37.488031 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:07:37.549536 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:07:37.593405 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:07:37.724818 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:37.738373 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:07:37.757867 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:38.081547 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:07:38.224735 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:38.241927 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:07:38.263228 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:07:38.332471 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:38.375826 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:07:38.401234 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:07:38.552034 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:07:38.663908 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:07:38.737777 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:07:38.759902 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:07:38.778838 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:38.827559 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:39.116848 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:07:39.195647 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:07:39.294129 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:07:39.298105 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:07:39.379140 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:39.390774 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
I20240501 14:07:39.416079 7071 maintenance_manager.cc:419] P 19a6d2d8a08142d5bffcde3a37234692: Scheduling FlushMRSOp(fee16905387d428aa02cd2070c86c790): perf score=0.033380
I20240501 14:07:39.459393 6962 maintenance_manager.cc:643] P 19a6d2d8a08142d5bffcde3a37234692: FlushMRSOp(fee16905387d428aa02cd2070c86c790) complete. Timing: real 0.043s user 0.029s sys 0.012s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":338,"dirs.run_cpu_time_us":155,"dirs.run_wall_time_us":1185,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":663,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":386,"threads_started":2,"wal-append.queue_time_us":269}
I20240501 14:07:39.460497 7071 maintenance_manager.cc:419] P 19a6d2d8a08142d5bffcde3a37234692: Scheduling UndoDeltaBlockGCOp(fee16905387d428aa02cd2070c86c790): 4834 bytes on disk
I20240501 14:07:39.461061 6962 maintenance_manager.cc:643] P 19a6d2d8a08142d5bffcde3a37234692: UndoDeltaBlockGCOp(fee16905387d428aa02cd2070c86c790) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4}
W20240501 14:07:39.461448 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:07:39.489740 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:07:39.491590 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 62 similar messages]
I20240501 14:07:39.517727 7334 maintenance_manager.cc:419] P 6dac90f01e744f9c8701a404440a4c77: Scheduling FlushMRSOp(23ca2bf1e59f44fcac8a48aff8d056b1): perf score=0.033371
I20240501 14:07:39.548823 7224 maintenance_manager.cc:643] P 6dac90f01e744f9c8701a404440a4c77: FlushMRSOp(23ca2bf1e59f44fcac8a48aff8d056b1) complete. Timing: real 0.031s user 0.021s sys 0.009s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":565,"dirs.run_cpu_time_us":197,"dirs.run_wall_time_us":1277,"drs_written":1,"lbm_read_time_us":45,"lbm_reads_lt_1ms":4,"lbm_write_time_us":576,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":638,"threads_started":2,"wal-append.queue_time_us":260}
I20240501 14:07:39.549988 7334 maintenance_manager.cc:419] P 6dac90f01e744f9c8701a404440a4c77: Scheduling UndoDeltaBlockGCOp(23ca2bf1e59f44fcac8a48aff8d056b1): 4834 bytes on disk
I20240501 14:07:39.550518 7224 maintenance_manager.cc:643] P 6dac90f01e744f9c8701a404440a4c77: UndoDeltaBlockGCOp(23ca2bf1e59f44fcac8a48aff8d056b1) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4}
I20240501 14:07:39.551326 7334 maintenance_manager.cc:419] P 6dac90f01e744f9c8701a404440a4c77: Scheduling FlushMRSOp(43b46fcf827c4275a47291afb107c290): perf score=0.033351
W20240501 14:07:39.559124 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:07:39.566900 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 73 similar messages]
I20240501 14:07:39.603018 7224 maintenance_manager.cc:643] P 6dac90f01e744f9c8701a404440a4c77: FlushMRSOp(43b46fcf827c4275a47291afb107c290) complete. Timing: real 0.052s user 0.030s sys 0.020s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":41,"dirs.run_cpu_time_us":233,"dirs.run_wall_time_us":1274,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":533,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":178,"threads_started":1,"wal-append.queue_time_us":278}
I20240501 14:07:39.604378 7334 maintenance_manager.cc:419] P 6dac90f01e744f9c8701a404440a4c77: Scheduling FlushMRSOp(ac4b234f794149e4b6def7a3b358942d): perf score=0.033342
I20240501 14:07:39.646893 7224 maintenance_manager.cc:643] P 6dac90f01e744f9c8701a404440a4c77: FlushMRSOp(ac4b234f794149e4b6def7a3b358942d) complete. Timing: real 0.042s user 0.037s sys 0.001s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":50,"dirs.run_cpu_time_us":189,"dirs.run_wall_time_us":1040,"drs_written":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4,"lbm_write_time_us":752,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":178,"threads_started":1,"wal-append.queue_time_us":272}
I20240501 14:07:39.648252 7334 maintenance_manager.cc:419] P 6dac90f01e744f9c8701a404440a4c77: Scheduling UndoDeltaBlockGCOp(ac4b234f794149e4b6def7a3b358942d): 4921 bytes on disk
I20240501 14:07:39.649295 7224 maintenance_manager.cc:643] P 6dac90f01e744f9c8701a404440a4c77: UndoDeltaBlockGCOp(ac4b234f794149e4b6def7a3b358942d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
I20240501 14:07:39.650079 7334 maintenance_manager.cc:419] P 6dac90f01e744f9c8701a404440a4c77: Scheduling UndoDeltaBlockGCOp(43b46fcf827c4275a47291afb107c290): 4833 bytes on disk
I20240501 14:07:39.650480 7224 maintenance_manager.cc:643] P 6dac90f01e744f9c8701a404440a4c77: UndoDeltaBlockGCOp(43b46fcf827c4275a47291afb107c290) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4}
W20240501 14:07:39.723297 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:07:39.877449 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:39.917629 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:07:39.923478 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:39.947537 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:07:40.148792 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 14:07:40.149679 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:07:40.244917 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:07:40.325356 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:40.381424 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:07:40.403542 7334 maintenance_manager.cc:419] P 6dac90f01e744f9c8701a404440a4c77: Scheduling FlushMRSOp(9b88c43cb8704583a3e65c5c35c205f6): perf score=0.033379
I20240501 14:07:40.433624 7224 maintenance_manager.cc:643] P 6dac90f01e744f9c8701a404440a4c77: FlushMRSOp(9b88c43cb8704583a3e65c5c35c205f6) complete. Timing: real 0.030s user 0.023s sys 0.006s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":285,"dirs.run_cpu_time_us":157,"dirs.run_wall_time_us":1310,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":532,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":420,"threads_started":2,"wal-append.queue_time_us":331}
I20240501 14:07:40.434726 7334 maintenance_manager.cc:419] P 6dac90f01e744f9c8701a404440a4c77: Scheduling UndoDeltaBlockGCOp(9b88c43cb8704583a3e65c5c35c205f6): 4921 bytes on disk
I20240501 14:07:40.435218 7224 maintenance_manager.cc:643] P 6dac90f01e744f9c8701a404440a4c77: UndoDeltaBlockGCOp(9b88c43cb8704583a3e65c5c35c205f6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":62,"lbm_reads_lt_1ms":4}
I20240501 14:07:40.464650 7071 maintenance_manager.cc:419] P 19a6d2d8a08142d5bffcde3a37234692: Scheduling FlushMRSOp(a726769d24284fedadd8e9e63cd27d05): perf score=0.033347
I20240501 14:07:40.493912 6962 maintenance_manager.cc:643] P 19a6d2d8a08142d5bffcde3a37234692: FlushMRSOp(a726769d24284fedadd8e9e63cd27d05) complete. Timing: real 0.029s user 0.024s sys 0.004s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":280,"dirs.run_cpu_time_us":172,"dirs.run_wall_time_us":1235,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":561,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"spinlock_wait_cycles":896,"thread_start_us":352,"threads_started":2,"wal-append.queue_time_us":258}
I20240501 14:07:40.494937 7071 maintenance_manager.cc:419] P 19a6d2d8a08142d5bffcde3a37234692: Scheduling UndoDeltaBlockGCOp(a726769d24284fedadd8e9e63cd27d05): 4921 bytes on disk
I20240501 14:07:40.495460 6962 maintenance_manager.cc:643] P 19a6d2d8a08142d5bffcde3a37234692: UndoDeltaBlockGCOp(a726769d24284fedadd8e9e63cd27d05) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4}
W20240501 14:07:40.708768 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:07:40.771595 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:40.806186 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:07:40.807086 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:07:40.827394 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:07:40.877907 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:07:40.911321 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
I20240501 14:07:40.998144 7071 maintenance_manager.cc:419] P 19a6d2d8a08142d5bffcde3a37234692: Scheduling FlushMRSOp(f9cb784f60774ea7b6e70ccfebf867a8): perf score=0.033370
W20240501 14:07:41.026909 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240501 14:07:41.038370 6962 maintenance_manager.cc:643] P 19a6d2d8a08142d5bffcde3a37234692: FlushMRSOp(f9cb784f60774ea7b6e70ccfebf867a8) complete. Timing: real 0.040s user 0.028s sys 0.010s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":7076,"dirs.run_cpu_time_us":228,"dirs.run_wall_time_us":1260,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":724,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":7351,"threads_started":2,"wal-append.queue_time_us":247}
I20240501 14:07:41.039351 7071 maintenance_manager.cc:419] P 19a6d2d8a08142d5bffcde3a37234692: Scheduling UndoDeltaBlockGCOp(f9cb784f60774ea7b6e70ccfebf867a8): 4921 bytes on disk
I20240501 14:07:41.039832 6962 maintenance_manager.cc:643] P 19a6d2d8a08142d5bffcde3a37234692: UndoDeltaBlockGCOp(f9cb784f60774ea7b6e70ccfebf867a8) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4}
W20240501 14:07:41.043555 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:07:41.125754 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:07:41.254323 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:41.287812 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
I20240501 14:07:41.291431 7071 maintenance_manager.cc:419] P 19a6d2d8a08142d5bffcde3a37234692: Scheduling FlushMRSOp(3d6859d53fd747a5a5e94722626fa522): perf score=0.033391
W20240501 14:07:41.325723 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240501 14:07:41.335839 6962 maintenance_manager.cc:643] P 19a6d2d8a08142d5bffcde3a37234692: FlushMRSOp(3d6859d53fd747a5a5e94722626fa522) complete. Timing: real 0.044s user 0.025s sys 0.016s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":44,"dirs.run_cpu_time_us":205,"dirs.run_wall_time_us":1198,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":916,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"spinlock_wait_cycles":3456,"thread_start_us":204,"threads_started":1,"wal-append.queue_time_us":326}
I20240501 14:07:41.336815 7071 maintenance_manager.cc:419] P 19a6d2d8a08142d5bffcde3a37234692: Scheduling UndoDeltaBlockGCOp(3d6859d53fd747a5a5e94722626fa522): 4921 bytes on disk
I20240501 14:07:41.337332 6962 maintenance_manager.cc:643] P 19a6d2d8a08142d5bffcde3a37234692: UndoDeltaBlockGCOp(3d6859d53fd747a5a5e94722626fa522) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
W20240501 14:07:41.389947 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:07:41.506975 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:07:41.592180 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:07:41.638180 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:07:41.642616 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:07:41.675923 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:41.812729 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:07:41.836724 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:41.927237 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:07:42.025068 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 14:07:42.191300 7334 maintenance_manager.cc:419] P 6dac90f01e744f9c8701a404440a4c77: Scheduling FlushMRSOp(420642af520d45ffbc7f2d35f93e75a7): perf score=0.033391
I20240501 14:07:42.220242 7224 maintenance_manager.cc:643] P 6dac90f01e744f9c8701a404440a4c77: FlushMRSOp(420642af520d45ffbc7f2d35f93e75a7) complete. Timing: real 0.029s user 0.026s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":297,"dirs.run_cpu_time_us":174,"dirs.run_wall_time_us":1240,"drs_written":1,"lbm_read_time_us":49,"lbm_reads_lt_1ms":4,"lbm_write_time_us":557,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":406,"threads_started":2,"wal-append.queue_time_us":283}
I20240501 14:07:42.221246 7334 maintenance_manager.cc:419] P 6dac90f01e744f9c8701a404440a4c77: Scheduling UndoDeltaBlockGCOp(420642af520d45ffbc7f2d35f93e75a7): 4579 bytes on disk
W20240501 14:07:42.227823 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
I20240501 14:07:42.232285 7224 maintenance_manager.cc:643] P 6dac90f01e744f9c8701a404440a4c77: UndoDeltaBlockGCOp(420642af520d45ffbc7f2d35f93e75a7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4}
I20240501 14:07:42.340930 7071 maintenance_manager.cc:419] P 19a6d2d8a08142d5bffcde3a37234692: Scheduling FlushMRSOp(2c39b53867e047faaf24cc6f87a8100c): perf score=0.033366
I20240501 14:07:42.373739 6962 maintenance_manager.cc:643] P 19a6d2d8a08142d5bffcde3a37234692: FlushMRSOp(2c39b53867e047faaf24cc6f87a8100c) complete. Timing: real 0.033s user 0.029s sys 0.000s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":295,"dirs.run_cpu_time_us":204,"dirs.run_wall_time_us":1261,"drs_written":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4,"lbm_write_time_us":587,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":382,"threads_started":2,"wal-append.queue_time_us":274}
I20240501 14:07:42.375064 7071 maintenance_manager.cc:419] P 19a6d2d8a08142d5bffcde3a37234692: Scheduling UndoDeltaBlockGCOp(2c39b53867e047faaf24cc6f87a8100c): 4581 bytes on disk
I20240501 14:07:42.375627 6962 maintenance_manager.cc:643] P 19a6d2d8a08142d5bffcde3a37234692: UndoDeltaBlockGCOp(2c39b53867e047faaf24cc6f87a8100c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4}
W20240501 14:07:42.384178 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:42.534744 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:07:42.596053 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:07:42.686097 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:07:42.690671 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:42.780650 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:07:42.782960 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:07:42.942397 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:43.199302 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:07:43.200268 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:07:43.290900 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:43.358698 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:07:43.369755 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:07:43.476555 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:07:43.548519 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:43.606995 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:07:43.613533 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:07:43.632148 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:07:43.871480 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:07:43.883054 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:43.984740 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
I20240501 14:07:43.988375 7334 maintenance_manager.cc:419] P 6dac90f01e744f9c8701a404440a4c77: Scheduling FlushMRSOp(1a7bbfc15a76450bbc9197037e856e08): perf score=0.033379
I20240501 14:07:44.024181 7224 maintenance_manager.cc:643] P 6dac90f01e744f9c8701a404440a4c77: FlushMRSOp(1a7bbfc15a76450bbc9197037e856e08) complete. Timing: real 0.036s user 0.020s sys 0.012s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":319,"dirs.run_cpu_time_us":181,"dirs.run_wall_time_us":1112,"drs_written":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4,"lbm_write_time_us":792,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":470,"threads_started":2,"wal-append.queue_time_us":328}
I20240501 14:07:44.025401 7334 maintenance_manager.cc:419] P 6dac90f01e744f9c8701a404440a4c77: Scheduling UndoDeltaBlockGCOp(1a7bbfc15a76450bbc9197037e856e08): 4921 bytes on disk
I20240501 14:07:44.025928 7224 maintenance_manager.cc:643] P 6dac90f01e744f9c8701a404440a4c77: UndoDeltaBlockGCOp(1a7bbfc15a76450bbc9197037e856e08) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4}
W20240501 14:07:44.064971 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:07:44.072872 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:44.203891 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:07:44.252447 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:07:44.287715 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:07:44.375387 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:07:44.499140 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:44.516479 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 14:07:44.517236 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:44.543812 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:07:44.575927 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 14:07:44.817355 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:07:44.917508 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:44.926111 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:44.975669 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:07:45.067371 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:07:45.116961 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:07:45.183296 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 14:07:45.305723 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:45.455312 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:45.531103 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:07:45.591701 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:07:45.746331 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:07:45.746551 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:07:45.775365 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:07:45.812350 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 14:07:45.812945 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:45.812914 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:07:45.835048 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:46.028959 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:07:46.195401 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:07:46.202328 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:07:46.310926 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 14:07:46.329660 21657 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:07:46.329816 21657 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:07:46.330684 21657 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers be56773105e744919601127a94e98adf (127.29.124.132:35459), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:07:46.330897 5386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf" is_pre_election: true
I20240501 14:07:46.331107 5386 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 2.
W20240501 14:07:46.332414 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:46.332577 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:07:46.332651 5191 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:07:46.332872 21657 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:07:46.338449 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:46.409394 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:46.486268 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:07:46.535964 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:07:46.741698 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:07:46.781396 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:07:46.861639 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:07:46.886493 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:46.921584 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:46.962199 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:07:47.333822 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:07:47.427539 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:47.460896 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:07:47.461433 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:07:47.507364 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:47.556754 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:07:47.571731 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:07:47.802537 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:07:47.883458 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:47.903461 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:47.996515 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:07:48.092721 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:07:48.258555 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:07:48.309166 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:07:48.353416 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:07:48.386466 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:48.498245 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:48.629495 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:07:48.675609 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:07:48.772689 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:07:48.791904 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:07:48.818452 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:48.820021 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:07:48.896270 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:48.972092 4840 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240501 14:07:48.973126 4840 catalog_manager.cc:4573] TS 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 8cc5eec4600640f7a293877c825f8d96: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:49.185983 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:07:49.218731 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:07:49.290369 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:07:49.366240 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:49.377843 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:07:49.409601 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:49.608034 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:07:49.683542 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240501 14:07:49.686635 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 14:07:49.811559 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:49.834244 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:49.881034 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:07:49.888908 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:07:49.929436 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:07:50.032088 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:07:50.039808 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:07:50.241370 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240501 14:07:50.319078 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:07:50.366492 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:50.390980 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:50.433436 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:07:50.490486 4840 catalog_manager.cc:4573] TS 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 1a7bbfc15a76450bbc9197037e856e08: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:50.580291 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:07:50.584354 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:07:50.738119 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:07:50.767587 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:50.854627 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:07:50.908866 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:51.007061 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 14:07:51.122776 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:07:51.130414 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:07:51.198066 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:07:51.313927 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:51.392729 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:07:51.399394 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:51.426235 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:07:51.561033 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:07:51.667588 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:07:51.757702 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:51.879715 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:52.022161 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:07:52.072140 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:07:52.153841 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:52.321180 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:07:52.363514 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:07:52.405764 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:52.426164 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:07:52.498224 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:07:52.596398 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:52.606177 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:07:52.641749 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:07:52.649552 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:07:52.956342 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:07:52.974821 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:53.007144 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:07:53.066795 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:53.092693 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:07:53.375944 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:07:53.412542 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:07:53.432535 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:53.612777 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:53.681062 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:07:53.718636 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:07:53.774410 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:07:53.776713 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:07:53.847401 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:07:54.015648 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:54.055075 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:07:54.074983 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:54.422704 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:07:54.432758 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:07:54.515497 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:54.519006 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:54.833480 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 14:07:54.834275 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:07:54.857017 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 14:07:54.868646 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:07:54.931748 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:07:54.952355 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:07:55.043529 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:07:55.062366 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:55.076854 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:55.096864 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:07:55.099259 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:07:55.224241 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:07:55.257193 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:07:55.333070 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 14:07:55.485635 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:55.648203 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:55.662514 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:07:55.786505 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:07:55.885936 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:07:55.920272 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:55.950206 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:07:55.959614 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:07:56.122283 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240501 14:07:56.139284 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:07:56.243464 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:56.283025 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
I20240501 14:07:56.313228 22504 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:07:56.313457 22504 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:07:56.314309 22504 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:07:56.314631 5255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" is_pre_election: true
I20240501 14:07:56.314813 5255 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 2.
W20240501 14:07:56.315728 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:56.315938 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:07:56.316025 5321 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:07:56.316254 22504 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:07:56.346508 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:07:56.396569 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:07:56.456717 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:07:56.482961 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:56.665113 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:56.842497 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:07:57.026248 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:57.111181 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:57.135362 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:07:57.207716 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:07:57.215122 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:07:57.506554 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:57.548911 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:07:57.557536 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:57.603164 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:07:57.607498 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:07:57.616075 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:07:57.751363 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:07:57.753964 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:07:57.818442 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:07:58.004365 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:58.029352 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:58.124534 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:07:58.243125 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:07:58.330287 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:07:58.349246 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:07:58.435791 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:07:58.438804 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:58.560820 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:58.643359 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:07:58.703202 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:07:58.840400 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:58.841403 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:07:58.902781 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:07:58.963485 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:07:59.035674 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:07:59.156451 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:59.374327 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:07:59.586323 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:07:59.603590 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:07:59.793510 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:07:59.836529 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 14:07:59.844448 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:07:59.911180 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240501 14:07:59.983072 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:00.106792 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:08:00.166559 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:08:00.205509 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:08:00.214469 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:00.218308 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:08:00.302485 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:08:00.405956 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 14:08:00.448797 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:08:00.454588 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:08:00.512130 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:08:00.543390 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:08:00.566977 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:08:00.586364 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:00.728387 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:00.785708 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:08:00.942413 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:08:01.050074 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:08:01.073801 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:01.141508 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:08:01.175773 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:08:01.334467 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:01.398461 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:08:01.494984 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:08:01.495287 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:08:01.500541 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:08:01.519115 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:01.918725 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:01.994455 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:02.103468 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:08:02.204923 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:08:02.415319 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:08:02.424127 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:08:02.444984 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:02.455065 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:08:02.604947 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:02.666292 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:08:02.752967 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:08:02.808287 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:08:02.821234 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:08:02.878367 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:08:02.951925 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:02.964660 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:08:02.970932 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:08:03.012506 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:03.017313 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:08:03.180881 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:08:03.281044 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:08:03.476197 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:08:03.521651 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:03.618114 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:03.626262 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:08:03.677677 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:08:03.785851 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:08:03.860109 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:08:04.037632 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:08:04.056746 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:04.143910 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:04.452885 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:04.528909 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:08:04.604529 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:08:04.636307 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
I20240501 14:08:04.854197 22268 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:08:04.854378 22268 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:08:04.855320 22268 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers be56773105e744919601127a94e98adf (127.29.124.132:35459), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:08:04.855615 5386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf" is_pre_election: true
I20240501 14:08:04.855852 5386 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 2.
W20240501 14:08:04.856163 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 14:08:04.857038 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:04.857196 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:08:04.857295 5191 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:08:04.857498 22268 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:08:04.934551 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 14:08:04.952033 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:04.979561 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:08:05.023758 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:05.076889 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:08:05.094147 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:08:05.103101 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:08:05.140056 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:08:05.192755 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:08:05.301609 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:08:05.307668 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:08:05.342832 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:08:05.358007 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:05.400722 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:05.500690 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:08:05.549962 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 14:08:05.658003 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:08:05.767226 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:08:05.783492 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:08:05.842161 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:05.857244 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:08:05.916429 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:05.921373 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:08:06.209681 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:08:06.210733 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:08:06.271615 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:08:06.279872 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:08:06.295449 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:06.427428 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:06.556658 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:08:06.725366 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:06.802695 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:08:06.909989 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:07.295156 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:07.389181 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:08:07.423872 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:08:07.528856 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:07.538879 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:08:07.571867 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:08:07.597848 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:08:07.631352 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:08:07.666800 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:08:07.702198 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:07.735508 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:08:07.745959 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:08:07.941504 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:08:07.945521 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:08.134835 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:08.151014 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:08:08.185796 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:08:08.216017 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:08:08.415180 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:08:08.458246 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:08:08.468379 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:08.495090 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:08:08.553498 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:08.714869 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:08:08.724778 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:08:08.776355 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:08:08.849624 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:08.973903 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:09.231489 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:09.339675 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:08:09.380872 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:08:09.460206 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:09.615543 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:09.877795 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:08:09.886263 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:08:09.951287 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 14:08:09.976632 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 14:08:10.015924 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:10.018232 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:08:10.067121 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:08:10.078452 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:08:10.170492 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:08:10.220685 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:10.229079 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:08:10.282810 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:08:10.362907 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:08:10.405243 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:08:10.451857 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:10.513983 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:08:10.558228 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240501 14:08:10.745973 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:08:10.748405 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:10.786921 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:08:10.882894 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:10.953680 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:08:11.062213 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:08:11.161688 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:11.177780 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:08:11.265156 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:08:11.304183 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:08:11.310442 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:08:11.398761 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:08:11.440248 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:11.662016 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:12.023604 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:08:12.040406 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:12.155148 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:08:12.179121 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:12.299537 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:08:12.319129 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:08:12.441181 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:08:12.468652 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:12.481493 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:08:12.618724 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:08:12.705375 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:08:12.727403 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:12.809782 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:08:12.894338 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:08:12.950858 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:12.995981 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:08:13.036542 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:08:13.251502 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:08:13.284129 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:13.327864 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:13.346318 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:08:13.377200 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:08:13.451431 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:08:13.643329 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:08:13.662441 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:08:13.670840 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:08:13.685163 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:08:13.825023 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:13.830705 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:13.870601 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:08:14.287813 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:14.336678 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:14.642709 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:08:14.738850 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:14.766109 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:08:14.811707 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:08:14.830404 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:08:14.861303 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:14.921505 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:08:14.993046 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240501 14:08:15.054034 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 14:08:15.065284 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:08:15.083509 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:08:15.154647 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:08:15.166853 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:08:15.348872 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:08:15.359931 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:15.389923 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:08:15.407176 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:15.504899 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:08:15.515460 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:08:15.759930 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:15.760639 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240501 14:08:15.828065 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:15.848645 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:08:15.938925 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:08:15.986044 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:08:16.022825 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:08:16.281697 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:08:16.289678 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:08:16.301630 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:16.340584 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:16.406014 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 14:08:16.491616 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:08:16.620676 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:08:16.851464 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:16.947966 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:16.996078 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:08:17.096591 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:08:17.246335 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:08:17.314190 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:17.533471 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:17.565416 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:08:17.590756 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:08:17.595232 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:08:17.779726 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 14:08:17.783413 24524 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:08:17.783581 24524 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:08:17.784422 24524 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers be56773105e744919601127a94e98adf (127.29.124.132:35459), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:08:17.784786 5386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf" is_pre_election: true
I20240501 14:08:17.785007 5386 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 2.
W20240501 14:08:17.786088 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:17.786239 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:08:17.786309 5191 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:08:17.786509 24524 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:08:17.834460 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:08:17.871840 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:08:17.879012 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:08:17.901523 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:08:17.911839 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:08:17.927409 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:17.950894 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:17.993824 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:08:18.357753 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:18.361088 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:18.399849 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:08:18.530718 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:08:18.588308 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:08:18.633039 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:08:18.697681 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:08:18.807297 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:08:18.841517 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:18.894203 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:19.040446 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:08:19.264704 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:08:19.274412 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:19.297435 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:19.751924 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:08:19.784251 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:19.784359 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:08:19.859704 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:19.950457 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:08:20.021445 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:08:20.104277 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:08:20.185441 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240501 14:08:20.186285 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:20.210062 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 14:08:20.210805 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:08:20.301090 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:08:20.306655 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:08:20.446686 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:20.456727 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:08:20.469717 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:08:20.523655 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:08:20.604549 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:08:20.629055 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:20.654621 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:08:20.738871 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:08:20.751422 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:08:20.895231 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:08:20.926793 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 14:08:20.927508 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:20.995955 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:08:21.244156 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:21.343773 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:08:21.349112 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:08:21.406749 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240501 14:08:21.414678 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:08:21.552520 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
I20240501 14:08:21.642588 24859 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:08:21.642786 24859 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:08:21.643687 24859 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers be56773105e744919601127a94e98adf (127.29.124.132:35459), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:08:21.644170 5386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf" is_pre_election: true
I20240501 14:08:21.644428 5386 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 2.
W20240501 14:08:21.645331 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:21.645493 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:08:21.645568 5191 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:08:21.645831 24859 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:08:21.668083 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:08:21.833765 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:22.113560 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:22.193359 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:08:22.263110 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:08:22.292256 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:08:22.305655 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:22.397847 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:08:22.405452 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:08:22.514685 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:08:22.677161 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:22.732894 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:08:22.833961 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:22.855000 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:08:22.925741 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:08:22.959933 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:08:23.014092 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:08:23.168260 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:08:23.224480 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:23.258136 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:08:23.312198 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:08:23.336293 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:08:23.341825 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:23.423324 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:08:23.459738 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:08:23.587074 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:08:23.799337 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:23.810624 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:24.034081 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:08:24.203138 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:08:24.261951 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:24.292768 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:24.472664 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:08:24.665635 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:24.691123 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:08:24.818518 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:08:24.896850 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:08:24.914217 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:24.963644 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:08:25.055966 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:08:25.090333 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:08:25.191965 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:08:25.195902 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240501 14:08:25.211462 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 14:08:25.216698 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:25.468449 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:25.483954 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:08:25.495966 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:08:25.547128 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:08:25.547171 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:08:25.616524 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:25.714215 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:08:25.726115 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:08:25.730079 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:08:25.769785 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:08:25.965854 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 14:08:25.975478 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:08:26.003664 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:26.020234 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:26.072629 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:08:26.231045 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:08:26.418751 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:26.539973 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:26.629900 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240501 14:08:26.642635 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:08:26.903286 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:08:26.997032 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:27.120362 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:27.170154 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:08:27.200978 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:08:27.293699 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:08:27.387463 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:08:27.444365 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:08:27.461658 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:27.490917 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:08:27.529531 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:08:27.653542 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:27.775293 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:08:27.788769 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:08:27.901643 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:08:27.932848 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:08:27.962126 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:08:28.027176 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:28.111184 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:28.153718 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:08:28.195183 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:08:28.266844 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:08:28.269474 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:08:28.480719 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:08:28.506909 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:28.610174 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:08:28.641870 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:28.679169 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:08:29.088565 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:29.207597 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:29.209615 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:08:29.248201 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:08:29.604823 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:08:29.639181 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:08:29.654034 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:29.793798 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:08:29.798900 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:29.922356 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:08:29.952394 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:08:30.051070 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:08:30.099226 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:08:30.144476 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:30.218081 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:30.220209 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 14:08:30.348400 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:08:30.406462 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240501 14:08:30.456416 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:08:30.499301 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:08:30.551324 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:08:30.568378 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:08:30.607400 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:08:30.608879 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:30.645058 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:08:30.743132 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
I20240501 14:08:30.765779 25677 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:08:30.765925 25677 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:08:30.766613 25677 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:08:30.766964 5255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" is_pre_election: true
I20240501 14:08:30.767199 5255 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 2.
W20240501 14:08:30.767874 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:30.767997 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:08:30.768049 5321 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:08:30.768234 25677 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:08:30.798804 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:30.865808 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:08:30.930580 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:08:31.020264 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:08:31.057405 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 14:08:31.124328 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:31.186875 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:31.229506 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:08:31.522940 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:31.699632 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:08:31.736498 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:08:31.765388 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:31.810918 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:08:31.982710 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:31.991235 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:08:32.197325 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:08:32.241114 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:32.255729 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:08:32.365751 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:32.487704 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:08:32.527217 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:08:32.591795 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:08:32.632174 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:08:32.805402 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:32.848829 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:08:32.981977 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:32.991797 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:08:33.038093 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:08:33.129006 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:08:33.133373 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:08:33.162732 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:08:33.292263 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:33.299943 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:08:33.367862 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:08:33.391172 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:08:33.397976 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:08:33.456619 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:33.544418 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:08:33.630673 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:08:33.746279 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
I20240501 14:08:33.964500 25782 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:08:33.964697 25782 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:08:33.965574 25782 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:08:33.966151 5255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" is_pre_election: true
I20240501 14:08:33.966378 5255 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 2.
W20240501 14:08:33.966939 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:33.967085 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:08:33.967154 5321 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:08:33.967366 25782 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:08:33.970166 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:08:33.986855 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:34.189710 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:08:34.207039 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:34.254601 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:08:34.395280 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:34.583863 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:34.755968 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:08:34.818871 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:08:34.880553 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:34.929585 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:08:35.028960 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:35.154058 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:08:35.223112 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240501 14:08:35.223888 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:08:35.306035 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:08:35.384577 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:08:35.472105 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:08:35.474936 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:08:35.481822 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:08:35.500593 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 14:08:35.501484 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:35.571257 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:35.647490 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:08:35.719326 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:08:35.797232 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:08:35.821348 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:08:35.864389 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:08:35.918663 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:35.922612 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:08:35.967767 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:35.977070 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:08:36.014819 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:08:36.202685 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 14:08:36.383038 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:36.383620 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:36.593291 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:08:36.733597 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:08:36.740374 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:08:36.811033 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:36.883210 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:37.007826 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:08:37.207700 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:37.244297 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:08:37.351295 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:37.444180 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:08:37.458017 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:08:37.540246 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:08:37.542516 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:08:37.745128 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:37.767510 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:37.852026 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:08:37.871752 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:08:37.884526 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:08:38.051504 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:08:38.103016 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:08:38.160063 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:08:38.212075 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:38.303131 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:08:38.329938 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:38.374658 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:08:38.421180 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:08:38.460482 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:08:38.464082 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:08:38.546505 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:08:38.607558 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:08:38.749119 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:38.887602 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:39.116561 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:08:39.175659 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:08:39.266198 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:39.383354 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:39.612007 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:08:39.794888 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:39.871888 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:39.920302 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:08:40.115324 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:08:40.116654 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:08:40.120189 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:08:40.140403 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:08:40.145956 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:08:40.165200 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:08:40.320932 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 14:08:40.368644 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:40.419914 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:40.444973 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:08:40.531903 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 14:08:40.563148 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:08:40.699666 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:08:40.792575 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:08:40.820896 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:08:40.825351 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:40.848446 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:08:40.888296 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:08:40.901953 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:08:40.942546 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:41.004186 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:08:41.184255 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:08:41.223433 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:08:41.256100 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240501 14:08:41.288717 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:41.455472 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:41.517549 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:08:41.538712 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:08:41.791229 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:41.807857 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:08:41.925099 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:42.174611 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:08:42.234700 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:42.238592 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:08:42.258667 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:08:42.539273 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:42.565532 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:08:42.628043 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:08:42.646564 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:08:42.667640 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:08:42.702334 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:42.749424 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:08:42.838109 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:08:42.992851 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:08:43.039768 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:08:43.102821 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:08:43.109004 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:43.143867 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:43.316095 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:08:43.378059 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:08:43.382398 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:08:43.591171 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:43.660276 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:43.670478 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:08:43.759660 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:08:43.853798 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:08:43.904680 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:08:43.921478 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:08:44.109699 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:08:44.172029 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:44.195698 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:44.630213 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:44.753314 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:08:44.782915 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:44.825150 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:08:44.863631 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:08:44.914048 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:08:45.042047 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:45.053496 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:08:45.115808 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:08:45.219548 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:45.284097 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:08:45.301541 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:08:45.427742 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:08:45.549610 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 14:08:45.550408 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:08:45.552351 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 14:08:45.582258 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:45.588860 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:08:45.658151 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:08:45.669276 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:08:45.729382 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:45.785192 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:08:45.961326 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:08:46.065856 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:46.140622 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:08:46.190784 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:08:46.280622 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 14:08:46.331915 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:46.459209 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:46.488415 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:08:46.539228 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:08:46.633421 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:08:46.785055 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:08:46.817042 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:46.908017 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240501 14:08:46.945899 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:47.243119 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:08:47.255107 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:47.338863 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:08:47.377415 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:08:47.501628 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:47.538831 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:08:47.549798 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:08:47.590160 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:08:47.749349 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:47.800365 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:08:47.801893 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:08:48.051565 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:08:48.068007 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:08:48.102638 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:48.133242 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:08:48.190973 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:08:48.258077 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:48.357832 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:08:48.375923 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:08:48.435528 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:08:48.580211 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:48.644202 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:48.747872 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:08:48.756887 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:08:48.988322 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:08:48.998078 4840 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240501 14:08:48.999063 4840 catalog_manager.cc:4573] TS 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 8cc5eec4600640f7a293877c825f8d96: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:49.016480 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:08:49.129294 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:49.180132 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:49.223711 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:08:49.505316 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:08:49.511384 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:08:49.537856 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:08:49.686033 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:49.689047 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:49.792529 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:08:49.888638 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:08:49.916764 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:08:50.172053 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:50.175930 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:08:50.223093 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:08:50.236228 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
I20240501 14:08:50.318027 27146 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:08:50.318183 27146 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:08:50.318979 27146 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:08:50.319370 5254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" is_pre_election: true
I20240501 14:08:50.319571 5254 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 2.
W20240501 14:08:50.320652 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:50.320803 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:08:50.320892 5321 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:08:50.321144 27146 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:08:50.375633 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:08:50.523527 4840 catalog_manager.cc:4573] TS 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 1a7bbfc15a76450bbc9197037e856e08: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:50.560684 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240501 14:08:50.561486 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:08:50.568149 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240501 14:08:50.568964 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:50.717089 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:08:50.768074 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:08:50.776985 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:08:50.815876 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:50.835574 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:08:50.882745 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:08:50.955197 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:08:51.123656 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:08:51.136973 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:51.256583 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:08:51.352720 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 14:08:51.406880 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:08:51.440380 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:51.476763 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:08:51.655565 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:51.791276 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:08:51.806309 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:08:51.892786 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:51.978763 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:08:51.998919 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:08:52.031971 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 14:08:52.165808 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:52.224668 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:08:52.294517 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:08:52.438349 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:08:52.472501 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:52.639122 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:08:52.662844 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:08:52.742997 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:52.925765 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:08:53.061322 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:53.099233 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:08:53.151252 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:08:53.153529 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:08:53.169853 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:53.201416 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:08:53.243381 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:08:53.342746 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:08:53.351503 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:08:53.551239 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:53.631296 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:08:53.668015 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:53.829296 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:08:53.950376 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:08:53.994418 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:08:54.075723 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:54.129417 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:08:54.168992 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:54.209523 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:08:54.406153 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:08:54.510718 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:54.639889 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:08:54.673610 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:54.699864 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
I20240501 14:08:54.787130 27759 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:08:54.787333 27759 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:08:54.788329 27759 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers be56773105e744919601127a94e98adf (127.29.124.132:35459), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:08:54.788944 5384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf" is_pre_election: true
I20240501 14:08:54.789170 5384 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 2.
W20240501 14:08:54.793941 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:54.794129 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:08:54.794215 5191 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:08:54.794512 27759 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:08:54.874722 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:08:54.916750 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:54.953640 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:08:55.069613 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:55.219391 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:08:55.334975 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:08:55.411775 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:55.465173 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:55.539094 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:08:55.563593 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:08:55.568475 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:08:55.589087 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 63 similar messages]
W20240501 14:08:55.601876 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 14:08:55.602859 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:08:55.637885 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:08:55.645160 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:08:55.931330 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:55.946918 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:55.954424 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:08:56.008502 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:08:56.058007 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:08:56.084576 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:08:56.353632 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:56.500900 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 14:08:56.505453 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:08:56.523573 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:08:56.532541 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:56.696257 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:08:56.725711 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:08:56.917431 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:57.055274 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:08:57.081329 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 14:08:57.094846 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:08:57.126585 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:57.154755 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:08:57.335768 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:57.443100 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:08:57.542178 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:57.599851 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:08:57.665827 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:08:57.766750 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:08:57.781090 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:57.949986 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:08:58.035638 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:08:58.088541 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:58.164304 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:08:58.181984 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:08:58.190441 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:08:58.191056 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:08:58.216861 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:58.231173 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:08:58.327963 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:08:58.607053 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:08:58.645687 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:58.653453 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:58.683945 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:08:58.879016 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:08:59.023375 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 14:08:59.029052 27146 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:08:59.029302 27146 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:08:59.030117 27146 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:08:59.030539 5254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" is_pre_election: true
I20240501 14:08:59.030800 5254 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 2.
W20240501 14:08:59.031404 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:59.031580 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:08:59.031667 5321 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:08:59.031939 27146 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:08:59.184293 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:59.223317 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:59.243942 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:08:59.249588 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:08:59.380501 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:08:59.409793 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:08:59.643517 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:08:59.794831 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:08:59.806810 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:08:59.911576 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:08:59.968364 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:08:59.980283 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:09:00.245801 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:00.308418 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:00.331809 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:09:00.342721 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:09:00.544536 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:09:00.581429 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:09:00.600076 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:09:00.689541 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 74 similar messages]
W20240501 14:09:00.690451 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:09:00.731925 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240501 14:09:00.732913 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:00.759282 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:09:00.803879 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:00.900939 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:09:00.907624 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:09:01.012694 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:09:01.120323 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:01.133810 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:09:01.285089 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:09:01.289427 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:01.366035 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:09:01.571769 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:01.599653 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 14:09:01.784577 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:01.795228 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:09:01.949723 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:09:01.952164 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:02.043015 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:09:02.045444 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:09:02.056823 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:09:02.092087 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:09:02.179535 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:02.438275 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:09:02.518874 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:02.610492 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:02.620811 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:09:02.670310 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:09:02.719880 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:09:02.977035 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:03.019399 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:09:03.027184 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:09:03.054069 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:09:03.142614 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:09:03.202792 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:09:03.208317 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:09:03.227181 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:03.302523 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:09:03.350032 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:09:03.438541 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:03.481467 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:09:03.587689 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:09:03.695396 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:09:03.705271 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:03.986590 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:04.086118 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:04.114567 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:09:04.235505 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:09:04.271853 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:09:04.384166 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:09:04.484807 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:04.504910 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:09:04.604297 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:04.619906 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:09:05.033938 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:05.081458 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:09:05.186762 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:09:05.209338 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:05.222713 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:09:05.404647 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:09:05.448719 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:09:05.510758 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:09:05.568022 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:05.622926 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:09:05.713335 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:09:05.715898 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:09:05.785843 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 14:09:05.821139 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 14:09:05.822357 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:05.827874 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:09:05.828670 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:09:05.869100 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:09:05.882099 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:09:05.931931 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:09:06.049711 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:06.204344 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:09:06.307138 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:06.480677 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:09:06.485777 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:09:06.591125 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:06.679366 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 14:09:06.920157 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:07.042740 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:09:07.121634 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 37 similar messages]
W20240501 14:09:07.122592 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:09:07.170995 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:07.272778 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:09:07.314337 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:07.355931 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:09:07.577670 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:09:07.593436 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:09:07.656246 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:09:07.716104 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:07.737223 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:07.752022 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:09:07.857812 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:09:08.004419 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:09:08.046303 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:09:08.157645 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:08.169631 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:09:08.248932 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:09:08.252230 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:09:08.348588 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:08.354265 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:09:08.450951 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:09:08.464226 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:09:08.663090 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:08.726141 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:09:08.812394 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:09:08.890983 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:09:08.934643 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:09:08.969213 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:09.236433 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:09.437217 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:09.572319 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:09:09.644899 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:09.715921 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:09:09.732828 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:09:09.874635 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:09:09.948582 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:10.105517 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:09:10.139060 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:09:10.183081 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:10.272637 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:09:10.313300 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:09:10.344211 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:09:10.404844 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:10.519008 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:09:10.595710 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:09:10.617576 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:10.668897 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:09:10.734717 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:09:10.822528 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 56 similar messages]
W20240501 14:09:10.849922 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 14:09:10.850879 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:09:10.858199 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:09:10.929668 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:09:10.970201 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:09:10.982792 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:10.992326 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:11.192277 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:09:11.382661 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:11.405952 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:09:11.491019 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:09:11.572568 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:11.667006 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:09:11.689898 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 14:09:11.984012 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:11.988996 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:12.112259 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:09:12.237064 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:09:12.309047 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:09:12.401844 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 14:09:12.412051 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:12.457100 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:09:12.547700 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:09:12.581445 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:09:12.599261 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:12.645630 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:09:12.775363 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:09:12.855999 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:12.953574 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:09:13.053591 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:09:13.071420 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:09:13.075012 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:09:13.192355 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:13.316947 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:09:13.370561 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:09:13.444180 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:13.456270 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:09:13.511317 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:09:13.580514 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:09:13.609766 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:09:13.644827 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:13.746750 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:09:13.942395 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:14.054358 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:09:14.092550 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:14.152630 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:09:14.344319 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:14.616524 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:09:14.651482 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:14.747486 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:14.793212 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:09:14.795976 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:09:14.818964 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:09:14.913967 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:09:15.049902 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:15.150100 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
I20240501 14:09:15.162312 29714 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:15.162547 29714 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:09:15.163625 29714 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers be56773105e744919601127a94e98adf (127.29.124.132:35459), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:09:15.163985 5386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf" is_pre_election: true
I20240501 14:09:15.164208 5386 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 2.
W20240501 14:09:15.165891 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:15.166077 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:09:15.166152 5191 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:09:15.166404 29714 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:09:15.202894 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:15.319525 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:09:15.373643 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:09:15.400970 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:09:15.530010 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:09:15.563045 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:15.663082 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:09:15.747738 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:09:15.803632 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:09:15.824242 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:15.849726 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:09:15.861196 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 64 similar messages]
W20240501 14:09:15.897828 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 14:09:15.898921 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:09:15.916126 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:09:15.931032 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:09:15.961179 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:16.156038 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:09:16.205150 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:16.208148 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:09:16.331573 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:09:16.545461 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:16.676780 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:09:16.704573 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 14:09:16.705547 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:16.919394 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:09:17.084199 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:17.092326 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:17.248135 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:09:17.279248 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:09:17.297906 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:09:17.492684 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:09:17.539053 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:17.547250 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:17.632594 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240501 14:09:17.639986 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:09:17.931816 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:09:17.944020 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:09:17.961131 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:09:17.992146 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:09:18.116845 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:09:18.130259 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:18.152940 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:18.186780 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:09:18.240615 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:09:18.338794 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:09:18.409896 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:09:18.462028 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:09:18.478505 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:09:18.511905 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:09:18.512471 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:18.616281 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:09:18.745877 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:19.008376 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:09:19.061306 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:19.191027 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:09:19.323529 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:09:19.350152 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:19.508739 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:19.523263 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:09:19.755012 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:09:19.797133 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:09:19.878402 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:09:19.950395 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:19.978448 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:20.165537 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:09:20.384693 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:20.396994 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:09:20.448892 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:09:20.463268 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:20.499917 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:09:20.536351 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:09:20.576005 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:09:20.710669 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:09:20.762073 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:09:20.842234 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:09:20.880295 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 56 similar messages]
W20240501 14:09:20.881165 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:20.938661 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:09:20.965608 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 14:09:20.966348 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:09:21.027390 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:09:21.044869 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:09:21.070607 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:21.315984 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:09:21.460937 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:21.507555 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:09:21.586398 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:09:21.686849 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:21.842147 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 14:09:21.846432 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:21.997664 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:09:22.103644 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:22.339991 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:09:22.366323 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:09:22.413898 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:22.429487 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:09:22.556537 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:22.594089 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:09:22.675072 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240501 14:09:22.769744 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:09:22.908102 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:09:22.974342 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:22.985625 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:23.018950 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:09:23.078312 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:09:23.181913 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:09:23.202622 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:09:23.218999 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:09:23.226080 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:09:23.228969 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:09:23.366114 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:09:23.385788 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:09:23.386287 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:09:23.392891 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:23.404001 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:09:23.449891 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:23.788945 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:09:23.801554 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:09:23.884634 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:23.926497 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:24.300072 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:09:24.417482 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:24.490214 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:24.520355 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:09:24.641547 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:09:24.860647 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:09:25.007318 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:09:25.009194 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:25.016996 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:09:25.074445 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:25.129606 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:09:25.349900 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:09:25.447288 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:09:25.449884 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:25.450353 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 14:09:25.532969 30751 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:25.533149 30751 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:09:25.534124 30751 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:09:25.534551 5255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" is_pre_election: true
I20240501 14:09:25.534770 5255 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 2.
W20240501 14:09:25.536995 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:25.537151 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:09:25.537232 5321 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:09:25.537660 30751 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:09:25.575500 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:09:25.595695 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:09:25.621035 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:09:25.657318 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:09:25.658942 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:25.750401 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:09:25.790916 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:09:25.855057 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:09:25.966045 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 14:09:25.966470 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:09:25.976688 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:25.983440 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240501 14:09:26.022938 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:09:26.212815 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:26.413304 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:09:26.421746 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:26.588616 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:09:26.762413 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:26.840570 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:26.899269 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 14:09:27.037422 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:09:27.051347 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:09:27.174552 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
I20240501 14:09:27.197834 30293 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:27.198014 30293 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:09:27.198791 30293 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers be56773105e744919601127a94e98adf (127.29.124.132:35459), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:09:27.199118 5386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf" is_pre_election: true
I20240501 14:09:27.199312 5386 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 2.
W20240501 14:09:27.200215 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:27.200418 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:09:27.200532 5191 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:09:27.200786 30293 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:09:27.271234 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:09:27.273736 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:27.346509 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:27.525635 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:09:27.656589 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:09:27.818624 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:09:27.878701 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:27.931061 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:28.026760 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240501 14:09:28.027617 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:09:28.032536 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:09:28.069175 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:09:28.089358 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:09:28.096078 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:09:28.112903 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:09:28.122346 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:09:28.131142 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:09:28.253139 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:09:28.345494 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:09:28.403052 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:28.434018 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:28.444586 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:09:28.486267 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:09:28.734872 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:09:28.834049 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:28.949126 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:09:29.002189 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:29.260113 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:29.410212 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:09:29.481796 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:29.603914 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:09:29.704963 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:09:29.711108 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:29.736436 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:09:29.913661 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:30.016451 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:09:30.299050 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:30.309213 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:09:30.385756 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:09:30.405170 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:09:30.428433 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:09:30.428982 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:09:30.443214 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:30.489745 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:09:30.557530 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:09:30.673354 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:09:30.686928 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:09:30.789480 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:30.795786 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:09:30.832687 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:09:30.853564 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:09:30.946408 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:09:30.972098 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240501 14:09:31.004001 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 14:09:31.023903 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:09:31.068276 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:31.138139 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:09:31.198570 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:31.399120 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:09:31.519065 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:31.583758 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:31.794766 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:09:31.804085 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:09:32.021997 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 14:09:32.035619 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:32.076893 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:32.275883 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:09:32.418253 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:09:32.422107 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:32.551491 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:09:32.595010 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:09:32.602423 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:32.814364 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:32.869536 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:09:32.888649 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:09:32.980320 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:09:33.020119 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:09:33.049933 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:09:33.092536 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:09:33.187008 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:09:33.187824 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:09:33.209373 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:33.256872 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:33.264681 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:09:33.303278 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:09:33.383627 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:09:33.511236 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:09:33.565708 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:09:33.600013 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:09:33.621639 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:09:33.740554 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:33.817974 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:34.029515 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:09:34.144966 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:09:34.259418 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:34.339318 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:09:34.363385 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:34.727483 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:34.813654 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:09:34.864881 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:34.937974 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:09:34.972555 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:09:35.122731 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:09:35.222523 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:35.228600 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:09:35.383819 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:09:35.432765 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:35.541148 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:09:35.623173 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:09:35.656255 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:09:35.669839 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:09:35.714046 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:35.782673 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:09:35.871131 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:35.924898 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:09:35.987869 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240501 14:09:35.988801 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:09:36.019778 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240501 14:09:36.045360 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:09:36.067163 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:09:36.069779 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:09:36.073760 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:09:36.217109 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:09:36.271950 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:36.316160 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
I20240501 14:09:36.319969 30966 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:36.320130 30966 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:09:36.320986 30966 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers be56773105e744919601127a94e98adf (127.29.124.132:35459), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:09:36.321316 5386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf" is_pre_election: true
I20240501 14:09:36.321549 5386 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 2.
W20240501 14:09:36.322250 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:36.322407 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:09:36.322492 5191 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:09:36.322733 30966 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:09:36.445228 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:09:36.821035 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:36.838613 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:36.962416 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:09:37.002904 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:09:37.070665 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240501 14:09:37.218762 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:37.328747 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:09:37.410988 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:37.487908 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:09:37.504910 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:09:37.509302 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:09:37.665045 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:37.667690 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:09:37.948601 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:37.979590 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:09:38.059237 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:09:38.118654 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:09:38.122841 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:09:38.155732 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:38.157814 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:09:38.255272 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:09:38.362301 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:38.407173 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:09:38.421453 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:09:38.422215 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:09:38.463882 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:09:38.504860 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:09:38.529826 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:09:38.538733 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:09:38.663430 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:38.901816 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:38.932469 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:09:39.004364 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:09:39.172127 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:39.360319 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:09:39.366284 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:39.643488 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:09:39.712759 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:39.713474 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:09:39.822355 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:39.859970 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:09:39.941701 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:09:40.126859 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:09:40.270920 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:40.377333 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
I20240501 14:09:40.443006 30966 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:40.443176 30966 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:09:40.443909 30966 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers be56773105e744919601127a94e98adf (127.29.124.132:35459), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:09:40.444298 5386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf" is_pre_election: true
I20240501 14:09:40.444572 5386 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 2.
W20240501 14:09:40.445655 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:40.445816 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:09:40.445880 5191 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:09:40.446125 30966 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:09:40.448699 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:09:40.492784 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:09:40.507755 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:09:40.556740 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:09:40.681845 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:40.703202 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:09:40.717561 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:09:40.771725 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:09:40.776484 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:09:40.779359 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:09:40.859866 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:40.882313 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:09:40.890071 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:09:40.921036 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:09:40.934311 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:09:41.098937 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 14:09:41.253537 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 65 similar messages]
W20240501 14:09:41.254485 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:41.399495 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:41.403767 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:09:41.636227 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:09:41.740115 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:09:41.820559 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:41.836076 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:41.971632 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:09:41.990116 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:09:42.084882 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:09:42.102883 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 14:09:42.272325 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:42.380743 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:42.447816 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:09:42.684151 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:09:42.690980 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:09:42.795332 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:42.880671 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:09:42.913916 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:42.930444 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:09:42.947216 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:09:43.213797 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:43.242780 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:09:43.280189 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:09:43.324955 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:09:43.349781 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:09:43.356411 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:09:43.421309 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:09:43.425861 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:09:43.447242 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:09:43.518308 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:43.598177 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:09:43.612550 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240501 14:09:43.639994 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:09:43.750694 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:44.086311 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:44.114825 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:09:44.166630 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:09:44.184640 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:44.360639 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:09:44.398845 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:09:44.554646 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:09:44.622694 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:44.624881 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:44.805459 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:09:45.055231 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:09:45.124457 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:09:45.146910 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:45.207401 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:09:45.210726 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:45.383692 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:09:45.436640 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:09:45.508129 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:09:45.553959 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:45.636454 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:09:45.652159 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:45.774147 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:09:45.857919 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:09:45.943545 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:09:46.014686 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:46.017235 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:09:46.040679 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:09:46.119976 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 14:09:46.133695 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:09:46.180368 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:09:46.223392 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:46.381922 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 14:09:46.382838 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:09:46.495424 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:46.609326 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:46.654788 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:09:46.719728 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:09:46.933050 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:09:46.960350 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:47.082180 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:09:47.133738 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:09:47.141294 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 14:09:47.142197 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:47.358604 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:47.389827 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:09:47.538681 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:09:47.612506 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:47.612852 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:09:47.795959 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:09:47.905140 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:09:47.929446 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:47.968362 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:09:47.987972 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:09:48.089121 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:48.118849 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:09:48.140244 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:09:48.247098 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:09:48.307854 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:48.310240 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:09:48.387259 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:09:48.658368 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:48.658519 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:09:48.693477 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 14:09:48.694525 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:09:48.720620 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:48.883589 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:09:49.014453 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:09:49.035765 4840 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20240501 14:09:49.036752 4840 catalog_manager.cc:4573] TS 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 8cc5eec4600640f7a293877c825f8d96: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:49.142294 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:49.213990 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:49.220284 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:09:49.242064 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:09:49.250640 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:09:49.542816 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:49.637284 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:49.728808 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:09:49.892984 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:09:49.921929 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:09:50.050123 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:09:50.059979 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:09:50.087790 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:09:50.138876 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:50.227870 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:50.409332 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:09:50.418747 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:09:50.496853 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:09:50.521138 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:50.555182 4840 catalog_manager.cc:4573] TS 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 1a7bbfc15a76450bbc9197037e856e08: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:50.608633 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:09:50.611384 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:09:50.757364 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:50.761471 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:09:51.005230 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:09:51.025985 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:09:51.045720 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
I20240501 14:09:51.050738 521 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:51.050892 521 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:09:51.051587 521 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers be56773105e744919601127a94e98adf (127.29.124.132:35459), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:09:51.051972 5386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf" is_pre_election: true
I20240501 14:09:51.052224 5386 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 2.
W20240501 14:09:51.053330 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:51.053485 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:09:51.053560 5191 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:09:51.053763 521 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:09:51.075306 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:51.100932 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:09:51.190268 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 67 similar messages]
W20240501 14:09:51.374995 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:51.440026 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240501 14:09:51.441143 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:09:51.599951 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:51.673211 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:09:51.771291 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:51.789953 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:09:51.975745 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:09:51.983412 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:52.024871 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:09:52.167249 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:09:52.204952 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:09:52.267472 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 14:09:52.268196 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:52.270184 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:09:52.514323 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:09:52.547016 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:52.578346 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:09:52.604724 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:09:52.767048 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:52.931918 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:09:53.007853 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:09:53.141830 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:53.157399 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:53.221597 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:09:53.300604 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:09:53.307269 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:09:53.325448 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:09:53.335973 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:09:53.401142 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:09:53.449393 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:09:53.618185 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:53.715242 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 37 similar messages]
W20240501 14:09:53.756999 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:09:53.778666 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
I20240501 14:09:54.077055 588 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:09:54.077308 588 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:09:54.078181 588 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:09:54.078706 5254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" is_pre_election: true
I20240501 14:09:54.078948 5254 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 2.
W20240501 14:09:54.079840 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:54.079988 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:09:54.080060 5321 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:09:54.080260 588 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:09:54.158653 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:54.159479 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:09:54.283186 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:54.354154 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:09:54.428915 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:09:54.508513 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:09:54.695837 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:54.735008 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:09:54.754032 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:09:54.805248 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:09:54.829172 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:09:54.850795 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:09:54.899622 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:54.990348 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:09:55.137941 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:09:55.193444 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:55.219054 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:09:55.462743 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:55.577718 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:55.679637 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:09:55.833948 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:09:55.843881 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:09:55.868340 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:09:55.947484 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:55.958653 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:09:55.998710 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:09:56.024953 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:56.045106 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:09:56.060796 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:09:56.268391 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:09:56.273492 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 14:09:56.483824 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:56.570734 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 14:09:56.571553 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:56.690954 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:09:56.856520 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:09:56.922159 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:56.950091 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:56.987149 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:09:57.078536 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:09:57.131990 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:09:57.222594 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:09:57.229502 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:09:57.265331 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:09:57.268760 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 14:09:57.304323 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:09:57.420121 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:09:57.444103 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:57.493566 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:57.665063 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:09:57.701445 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:09:57.884781 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:57.936204 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:58.257041 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:09:58.261837 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:09:58.301679 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:09:58.379962 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:58.420258 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:09:58.458609 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:58.487422 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:09:58.490300 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:09:58.588865 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:09:58.633132 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:09:58.916261 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:09:58.926081 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:09:58.943145 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:59.010783 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:59.015784 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:09:59.446792 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:09:59.516583 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:09:59.555145 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:09:59.561105 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:09:59.630193 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:09:59.691781 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:09:59.724810 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:09:59.818683 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:09:59.914541 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:09:59.925156 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:09:59.935739 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:09:59.955902 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:00.024263 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:00.174829 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:10:00.313467 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:10:00.421888 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:00.478042 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:10:00.592947 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:00.761006 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:10:00.843825 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:10:00.879276 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:10:00.928275 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:10:00.930855 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:10:00.951231 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:01.105855 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:10:01.151613 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:10:01.174963 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:01.222679 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:10:01.308459 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240501 14:10:01.387615 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:01.720227 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 14:10:01.768987 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:10:01.779449 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:01.871423 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:01.900350 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:10:02.054841 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:10:02.104161 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:10:02.117877 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:10:02.274274 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 14:10:02.275257 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:02.298498 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:10:02.304421 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:10:02.356711 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:10:02.385365 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:02.517210 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:10:02.614490 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:10:02.760404 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:02.765528 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
I20240501 14:10:02.810163 1687 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:02.810323 1687 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:10:02.811118 1687 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers be56773105e744919601127a94e98adf (127.29.124.132:35459), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:10:02.811547 5386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf" is_pre_election: true
I20240501 14:10:02.811769 5386 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 2.
W20240501 14:10:02.812784 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:02.812960 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:10:02.813041 5191 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:10:02.813284 1687 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:10:02.896430 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:02.910655 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:10:03.004747 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:10:03.005012 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:10:03.279568 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:03.335532 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:03.366039 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:10:03.380029 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:10:03.435182 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:10:03.505576 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:10:03.583410 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:10:03.733636 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:10:03.835851 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:03.874742 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:04.039415 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 14:10:04.099750 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:10:04.319079 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:10:04.419736 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:04.457597 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:04.502521 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:10:04.547389 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:10:04.593408 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:10:04.685325 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:10:04.863113 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:10:04.864056 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:10:04.880720 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:10:04.926728 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:04.930924 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:05.164005 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:10:05.166800 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:10:05.184075 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:10:05.299350 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:10:05.347496 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:10:05.395423 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:05.474296 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:05.514353 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:10:05.764636 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:10:05.822504 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:05.973418 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:10:06.030174 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:10:06.048111 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:06.086526 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:10:06.102666 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:10:06.204241 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:10:06.311473 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 67 similar messages]
W20240501 14:10:06.427595 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:06.621505 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:06.672205 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:10:06.937872 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 64 similar messages]
W20240501 14:10:06.938946 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:10:06.942171 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:06.986435 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:10:07.020160 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:10:07.050551 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:07.058734 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:10:07.145364 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:10:07.158465 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:10:07.214479 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:10:07.276028 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 14:10:07.469542 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:07.511291 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:10:07.543052 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:10:07.547318 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:10:07.642918 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:07.643051 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:10:07.767556 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:10:07.844262 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:10:07.869088 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:08.050871 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:08.079748 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:10:08.134281 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:10:08.410926 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:08.443837 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:10:08.540648 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:08.544931 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:10:08.569931 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:10:08.706136 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:10:08.858215 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:10:08.898883 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:09.089850 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:10:09.164064 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:09.173207 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 14:10:09.353070 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 14:10:09.426467 2252 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:09.426658 2252 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:10:09.427419 2252 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers be56773105e744919601127a94e98adf (127.29.124.132:35459), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:10:09.427939 5386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf" is_pre_election: true
I20240501 14:10:09.428210 5386 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 2.
W20240501 14:10:09.429186 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:09.429384 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:10:09.429478 5191 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:10:09.429718 2252 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:10:09.475598 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:10:09.479756 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:10:09.498045 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:09.600276 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:10:09.613060 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:09.617048 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:10:09.644784 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:10:09.726795 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:10:09.740572 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:10:09.857641 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:10:09.870697 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:10:09.978809 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:10.206246 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:10.279655 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:10:10.297176 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:10:10.396263 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:10:10.566417 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:10:10.600998 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:10.741712 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:10.811635 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:10:11.031200 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:10:11.050968 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:11.123546 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:10:11.154210 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:11.157339 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:10:11.226961 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:10:11.411478 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 14:10:11.443431 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:10:11.518288 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:11.651306 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:11.752702 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:10:11.861930 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:10:11.925621 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:10:12.035773 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:10:12.106366 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:12.122747 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 63 similar messages]
W20240501 14:10:12.123828 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:12.129072 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:10:12.129695 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:10:12.174955 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:10:12.246999 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:10:12.258080 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:10:12.296564 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 14:10:12.378444 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:10:12.470618 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:10:12.524159 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:12.593717 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:10:12.601490 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:12.712950 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:10:12.792856 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:10:12.858899 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:10:12.902356 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:13.019670 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:10:13.194113 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:13.397451 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:13.432277 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:10:13.703887 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:13.717764 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:10:13.782831 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:10:13.805063 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:10:13.979934 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:14.116160 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:10:14.149726 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:10:14.295702 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:14.298830 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240501 14:10:14.460014 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:14.508065 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:10:14.556321 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:10:14.622208 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:10:14.644790 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:10:14.661141 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:10:14.730742 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:14.858405 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:10:14.881516 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:10:14.884836 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:10:14.891841 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:14.979880 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:10:15.048195 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:10:15.092895 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:10:15.126973 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:10:15.178607 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:15.185315 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:10:15.268850 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:10:15.299468 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:15.326859 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:10:15.717634 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:15.921479 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:16.087270 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:10:16.146852 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:16.206041 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:10:16.396821 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:10:16.452208 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:16.469871 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 14:10:16.508381 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:10:16.559394 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:10:16.587224 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:10:16.665051 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:16.978883 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:10:17.002328 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:10:17.026468 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:17.090952 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:10:17.132839 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 59 similar messages]
W20240501 14:10:17.160924 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:10:17.170439 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:17.202257 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:10:17.215945 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:10:17.218120 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:10:17.347910 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:10:17.348085 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:10:17.535385 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 14:10:17.545336 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:17.583055 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:10:17.624644 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:17.629113 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:10:17.709648 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:10:17.750231 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:10:17.788146 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:10:17.888448 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:10:17.979475 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:18.064601 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:18.466945 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:18.490759 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:10:18.573751 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:10:18.584142 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:18.860834 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:18.880007 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:10:18.987228 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:10:18.987344 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:18.991726 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:10:19.043697 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:10:19.434314 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:10:19.471607 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:19.474979 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:19.514811 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:10:19.594980 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:10:19.671185 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:10:19.676268 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:10:19.692535 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:10:19.695591 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:10:19.727903 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:10:19.876448 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:10:19.886029 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:10:19.966503 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:20.047163 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:10:20.057377 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:20.094552 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:10:20.175566 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:10:20.197660 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:10:20.416628 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:10:20.487154 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:20.559860 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:10:20.627072 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:20.938388 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:10:20.964090 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:21.136305 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:10:21.149942 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:21.228577 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:10:21.416882 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:10:21.506600 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:10:21.547688 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240501 14:10:21.553596 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:10:21.562981 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:21.574553 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:21.957682 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:10:22.018342 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:10:22.028816 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:22.146872 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:10:22.147888 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:22.166364 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 14:10:22.185897 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:10:22.219892 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:10:22.379710 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:10:22.409222 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:10:22.412912 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:10:22.460515 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:10:22.497030 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:22.568195 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 61 similar messages]
W20240501 14:10:22.569312 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:10:22.577467 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:10:22.658644 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:22.771114 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:10:22.798599 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:10:22.887526 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:10:23.077759 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:23.223112 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:23.410946 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:10:23.415813 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:10:23.609676 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:10:23.609757 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:23.632722 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:23.657850 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:10:23.692322 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:10:23.878731 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:10:24.043026 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
I20240501 14:10:24.056674 3630 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:24.056864 3630 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:10:24.057718 3630 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:10:24.058038 5254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" is_pre_election: true
I20240501 14:10:24.058228 5254 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 2.
W20240501 14:10:24.059091 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:24.059267 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:10:24.059350 5321 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:10:24.059553 3630 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:10:24.090481 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:24.163213 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:24.433360 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:10:24.537796 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:24.557837 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:10:24.590471 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240501 14:10:24.699168 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:10:24.708222 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:24.764815 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:10:24.819243 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:10:24.844894 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:10:24.962255 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:10:24.981479 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:10:25.000211 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:10:25.019740 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:25.121599 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:10:25.121898 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:10:25.136441 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:10:25.165341 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:25.371752 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:10:25.462181 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:25.672389 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:25.701097 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:10:25.771224 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:10:25.911841 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:26.099699 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:10:26.126173 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:26.135663 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:10:26.275164 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:10:26.341586 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:10:26.433315 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:26.490603 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:10:26.618187 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:26.647111 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240501 14:10:26.811026 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:10:26.854008 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:26.879179 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:10:27.188146 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:10:27.224010 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:27.255056 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:10:27.298789 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:10:27.301052 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 58 similar messages]
W20240501 14:10:27.301856 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:27.308943 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:10:27.330909 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:10:27.391120 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:10:27.446553 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:10:27.626960 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 57 similar messages]
W20240501 14:10:27.650874 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:10:27.659574 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:10:27.699498 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:27.747875 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:27.906052 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:10:27.937018 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:10:28.048931 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:10:28.102488 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:28.175311 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:10:28.314105 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:10:28.336982 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:28.497511 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:10:28.562103 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:10:28.572721 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:10:28.652478 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:28.717784 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:28.954509 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:10:28.987507 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:10:29.028988 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:29.340226 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:29.383975 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:10:29.480887 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:10:29.526686 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:10:29.530764 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:29.602362 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:10:29.702991 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:10:29.710098 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:10:29.719200 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:10:29.836733 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:10:29.853824 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:10:29.872393 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:29.961395 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:30.038076 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:10:30.126891 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:10:30.297838 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:30.373993 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:10:30.442335 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:10:30.468677 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:10:30.475666 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:30.490832 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:10:30.591596 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:10:30.628158 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:10:30.790993 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:30.948423 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:31.027623 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:10:31.057940 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:10:31.099934 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240501 14:10:31.275826 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:31.303325 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:10:31.361202 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:10:31.495466 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:31.697088 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:10:31.791592 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240501 14:10:31.802443 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:31.887944 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:32.010874 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:10:32.046597 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:10:32.192548 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:10:32.217931 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:10:32.219408 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:10:32.226127 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:10:32.247089 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:32.290517 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:10:32.313928 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:10:32.315512 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240501 14:10:32.316371 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:10:32.334697 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:32.706327 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 62 similar messages]
W20240501 14:10:32.729002 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:32.769539 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:32.865073 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:10:32.942435 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:10:33.014379 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:10:33.072268 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:10:33.125968 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:10:33.136849 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:10:33.187302 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:33.344250 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:33.539755 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:10:33.544031 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:10:33.595990 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240501 14:10:33.625905 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:33.744801 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:10:33.746793 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:10:33.783564 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:34.117818 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240501 14:10:34.190568 4461 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:34.190749 4461 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:10:34.191818 4461 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers be56773105e744919601127a94e98adf (127.29.124.132:35459), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:10:34.192103 5386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf" is_pre_election: true
I20240501 14:10:34.192310 5386 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 2.
W20240501 14:10:34.193621 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:34.193806 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:10:34.193897 5191 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:10:34.194082 4461 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:10:34.249884 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:34.280455 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:10:34.365423 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:34.428025 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:10:34.488013 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:10:34.546214 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:10:34.630023 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:10:34.678282 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240501 14:10:34.770926 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:10:34.855702 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:34.875460 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:10:34.893785 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:10:34.923959 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:35.036176 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:10:35.280612 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:35.414356 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:10:35.425485 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:10:35.491900 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:35.499231 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:10:35.512684 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:10:35.604422 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:10:35.625342 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:10:35.793491 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:35.869274 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:35.997355 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:10:35.997439 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:10:36.026942 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:10:36.193552 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:10:36.244711 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:36.291872 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240501 14:10:36.312309 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:36.493705 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:10:36.788107 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:10:36.803373 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:36.871380 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:36.948617 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240501 14:10:36.949512 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:10:36.978399 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:10:36.980232 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:10:37.027298 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:10:37.189014 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:10:37.239996 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:37.378491 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 63 similar messages]
W20240501 14:10:37.391374 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:10:37.483016 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:37.573531 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:10:37.586486 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:10:37.717994 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:10:37.785653 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 60 similar messages]
W20240501 14:10:37.786372 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:10:37.807746 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:10:37.809423 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:37.878319 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:38.073331 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:10:38.164178 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:10:38.191203 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:10:38.224010 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:10:38.228928 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:38.414019 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:10:38.467033 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:38.509110 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:10:38.619462 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:10:38.682758 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:38.873391 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240501 14:10:39.038518 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 14:10:39.056313 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:39.267203 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:39.395272 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 14:10:39.404795 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:10:39.409984 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:10:39.455324 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 14:10:39.495152 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:39.594650 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:10:39.698349 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:10:39.746928 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:39.885577 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240501 14:10:39.936237 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 14:10:39.986809 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 14:10:39.991091 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:40.060768 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:10:40.316581 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:40.367938 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 14:10:40.423466 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:40.511719 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 14:10:40.584128 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:10:40.694729 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 14:10:40.706905 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 14:10:40.726734 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 14:10:40.766795 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:10:40.898313 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:40.917938 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:40.937376 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:10:40.946048 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240501 14:10:40.980790 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 14:10:41.313187 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:41.451972 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240501 14:10:41.459981 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:41.493276 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
I20240501 14:10:41.691471 5794 consensus_queue.cc:579] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Leader has been unable to successfully communicate with peer 4360b59e3a154688a79ca5f9c0a37bc9 for more than 300 seconds (300.019s)
W20240501 14:10:41.693549 5061 consensus_peers.cc:462] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
I20240501 14:10:41.709512 5125 consensus_queue.cc:237] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } }
I20240501 14:10:41.711717 5384 raft_consensus.cc:1261] T 9d32e0707d244f2c9bbcb57f62b09f31 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:10:41.712244 7285 raft_consensus.cc:1261] T 9d32e0707d244f2c9bbcb57f62b09f31 P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:10:41.712204 5893 consensus_queue.cc:1035] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:10:41.712764 5893 consensus_queue.cc:1035] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:10:41.715744 5892 raft_consensus.cc:2932] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:10:41.718683 7285 raft_consensus.cc:2932] T 9d32e0707d244f2c9bbcb57f62b09f31 P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:10:41.721721 4839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9d32e0707d244f2c9bbcb57f62b09f31 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:10:41.722841 4850 catalog_manager.cc:5484] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 reported cstate change: config changed from index 7 to 8, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) evicted. New cstate: current_term: 1 leader_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:10:41.727165 5384 raft_consensus.cc:2932] T 9d32e0707d244f2c9bbcb57f62b09f31 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
W20240501 14:10:41.731527 4840 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 1 similar messages]
I20240501 14:10:41.731874 5125 consensus_queue.cc:237] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: NON_VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { promote: true } }
W20240501 14:10:41.732731 4840 catalog_manager.cc:4573] TS 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9d32e0707d244f2c9bbcb57f62b09f31: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
I20240501 14:10:41.734349 5384 raft_consensus.cc:1261] T 9d32e0707d244f2c9bbcb57f62b09f31 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:10:41.734529 7285 raft_consensus.cc:1261] T 9d32e0707d244f2c9bbcb57f62b09f31 P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:10:41.734925 5794 consensus_queue.cc:1035] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240501 14:10:41.735231 5059 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209): Couldn't send request to peer 1c3fdc5272484ac09ef3e5e36a08f27c. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9d32e0707d244f2c9bbcb57f62b09f31. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:41.735420 5893 consensus_queue.cc:1035] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:10:41.737336 5893 raft_consensus.cc:2932] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: NON_VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { promote: true } } }
W20240501 14:10:41.737864 5321 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 14:10:41.737742 5384 raft_consensus.cc:2932] T 9d32e0707d244f2c9bbcb57f62b09f31 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: NON_VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { promote: true } } }
I20240501 14:10:41.738324 7285 raft_consensus.cc:2932] T 9d32e0707d244f2c9bbcb57f62b09f31 P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: NON_VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { promote: true } } }
I20240501 14:10:41.741189 4839 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9d32e0707d244f2c9bbcb57f62b09f31 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:10:41.743254 4847 catalog_manager.cc:5484] T 9d32e0707d244f2c9bbcb57f62b09f31 P 6dac90f01e744f9c8701a404440a4c77 reported cstate change: config changed from index 8 to 9, NON_VOTER 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) added. New cstate: current_term: 1 leader_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: NON_VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { promote: true } } }
W20240501 14:10:41.835208 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 14:10:41.862447 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:41.961853 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240501 14:10:41.962704 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Error code: TABLET_NOT_RUNNING (12). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240501 14:10:42.054752 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240501 14:10:42.087582 4656 consensus_queue.cc:579] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Leader has been unable to successfully communicate with peer 4360b59e3a154688a79ca5f9c0a37bc9 for more than 300 seconds (300.081s)
I20240501 14:10:42.097353 4993 consensus_queue.cc:237] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: NON_VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { promote: true } }
I20240501 14:10:42.099797 5384 raft_consensus.cc:1261] T 99d958617b7e4ef385b94f4c73f4dd84 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 14:10:42.099892 4928 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209): Couldn't send request to peer 1c3fdc5272484ac09ef3e5e36a08f27c. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:42.100298 5930 consensus_queue.cc:1035] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 14:10:42.101207 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:42.102933 4656 raft_consensus.cc:2932] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: NON_VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { promote: true } } }
I20240501 14:10:42.103564 5384 raft_consensus.cc:2932] T 99d958617b7e4ef385b94f4c73f4dd84 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: NON_VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { promote: true } } }
I20240501 14:10:42.106231 4839 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 99d958617b7e4ef385b94f4c73f4dd84 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:10:42.109061 4852 catalog_manager.cc:5484] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 reported cstate change: config changed from index 6 to 7, NON_VOTER 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) added. New cstate: current_term: 1 leader_uuid: "04088ee3bea440a7bd5ff772152b9410" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: NON_VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 14:10:42.111506 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240501 14:10:42.165405 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 14:10:42.220890 5930 consensus_queue.cc:579] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Leader has been unable to successfully communicate with peer 4360b59e3a154688a79ca5f9c0a37bc9 for more than 300 seconds (300.061s)
I20240501 14:10:42.224493 4993 consensus_queue.cc:237] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } }
I20240501 14:10:42.226728 5254 raft_consensus.cc:1261] T ba6cba753e934e719c730593c0706346 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:10:42.227197 5935 consensus_queue.cc:1035] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 14:10:42.227468 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ba6cba753e934e719c730593c0706346. This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:42.227990 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:42.229606 5931 raft_consensus.cc:2932] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } }
I20240501 14:10:42.230157 5254 raft_consensus.cc:2932] T ba6cba753e934e719c730593c0706346 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } }
I20240501 14:10:42.232539 4839 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ba6cba753e934e719c730593c0706346 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:10:42.234298 4850 catalog_manager.cc:5484] T ba6cba753e934e719c730593c0706346 P 1c3fdc5272484ac09ef3e5e36a08f27c reported cstate change: config changed from index 6 to 7, NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) added. New cstate: current_term: 1 leader_uuid: "04088ee3bea440a7bd5ff772152b9410" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } }
I20240501 14:10:42.244763 5962 ts_tablet_manager.cc:902] T 9d32e0707d244f2c9bbcb57f62b09f31 P 1c3fdc5272484ac09ef3e5e36a08f27c: Initiating tablet copy from peer e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377)
I20240501 14:10:42.245803 5962 tablet_copy_client.cc:308] T 9d32e0707d244f2c9bbcb57f62b09f31 P 1c3fdc5272484ac09ef3e5e36a08f27c: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.130:34377
I20240501 14:10:42.246455 5146 tablet_copy_service.cc:140] P e9c60449cf3c46f8aca4ab6c535302a4: Received BeginTabletCopySession request for tablet 9d32e0707d244f2c9bbcb57f62b09f31 from peer 1c3fdc5272484ac09ef3e5e36a08f27c ({username='slave'} at 127.29.124.131:42095)
I20240501 14:10:42.246604 5146 tablet_copy_service.cc:161] P e9c60449cf3c46f8aca4ab6c535302a4: Beginning new tablet copy session on tablet 9d32e0707d244f2c9bbcb57f62b09f31 from peer 1c3fdc5272484ac09ef3e5e36a08f27c at {username='slave'} at 127.29.124.131:42095: session id = 1c3fdc5272484ac09ef3e5e36a08f27c-9d32e0707d244f2c9bbcb57f62b09f31
I20240501 14:10:42.249305 5146 tablet_copy_source_session.cc:205] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 14:10:42.250784 5962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9d32e0707d244f2c9bbcb57f62b09f31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:42.255002 5962 tablet_copy_client.cc:783] T 9d32e0707d244f2c9bbcb57f62b09f31 P 1c3fdc5272484ac09ef3e5e36a08f27c: tablet copy: Starting download of 5 data blocks...
I20240501 14:10:42.260733 5962 tablet_copy_client.cc:647] T 9d32e0707d244f2c9bbcb57f62b09f31 P 1c3fdc5272484ac09ef3e5e36a08f27c: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:42.262218 5962 tablet_copy_client.cc:523] T 9d32e0707d244f2c9bbcb57f62b09f31 P 1c3fdc5272484ac09ef3e5e36a08f27c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:42.264899 5962 tablet_bootstrap.cc:492] T 9d32e0707d244f2c9bbcb57f62b09f31 P 1c3fdc5272484ac09ef3e5e36a08f27c: Bootstrap starting.
I20240501 14:10:42.277638 5962 tablet_bootstrap.cc:492] T 9d32e0707d244f2c9bbcb57f62b09f31 P 1c3fdc5272484ac09ef3e5e36a08f27c: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:42.277944 5962 tablet_bootstrap.cc:492] T 9d32e0707d244f2c9bbcb57f62b09f31 P 1c3fdc5272484ac09ef3e5e36a08f27c: Bootstrap complete.
I20240501 14:10:42.278167 5962 ts_tablet_manager.cc:1367] T 9d32e0707d244f2c9bbcb57f62b09f31 P 1c3fdc5272484ac09ef3e5e36a08f27c: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.004s
I20240501 14:10:42.278788 5962 raft_consensus.cc:348] T 9d32e0707d244f2c9bbcb57f62b09f31 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: NON_VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { promote: true } }
I20240501 14:10:42.279038 5962 raft_consensus.cc:729] T 9d32e0707d244f2c9bbcb57f62b09f31 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1c3fdc5272484ac09ef3e5e36a08f27c, State: Initialized, Role: LEARNER
I20240501 14:10:42.279253 5962 consensus_queue.cc:260] T 9d32e0707d244f2c9bbcb57f62b09f31 P 1c3fdc5272484ac09ef3e5e36a08f27c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: NON_VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { promote: true } }
I20240501 14:10:42.279742 5962 ts_tablet_manager.cc:1398] T 9d32e0707d244f2c9bbcb57f62b09f31 P 1c3fdc5272484ac09ef3e5e36a08f27c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:10:42.280377 5146 tablet_copy_service.cc:342] P e9c60449cf3c46f8aca4ab6c535302a4: Request end of tablet copy session 1c3fdc5272484ac09ef3e5e36a08f27c-9d32e0707d244f2c9bbcb57f62b09f31 received from {username='slave'} at 127.29.124.131:42095
I20240501 14:10:42.280483 5146 tablet_copy_service.cc:433] P e9c60449cf3c46f8aca4ab6c535302a4: ending tablet copy session 1c3fdc5272484ac09ef3e5e36a08f27c-9d32e0707d244f2c9bbcb57f62b09f31 on tablet 9d32e0707d244f2c9bbcb57f62b09f31 with peer 1c3fdc5272484ac09ef3e5e36a08f27c
W20240501 14:10:42.351773 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240501 14:10:42.412510 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 59 similar messages]
I20240501 14:10:42.477336 5727 consensus_queue.cc:579] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf [LEADER]: Leader has been unable to successfully communicate with peer 4360b59e3a154688a79ca5f9c0a37bc9 for more than 300 seconds (300.099s)
I20240501 14:10:42.481845 5302 maintenance_manager.cc:419] P 1c3fdc5272484ac09ef3e5e36a08f27c: Scheduling UndoDeltaBlockGCOp(9d32e0707d244f2c9bbcb57f62b09f31): 4581 bytes on disk
I20240501 14:10:42.482467 5193 maintenance_manager.cc:643] P 1c3fdc5272484ac09ef3e5e36a08f27c: UndoDeltaBlockGCOp(9d32e0707d244f2c9bbcb57f62b09f31) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4}
I20240501 14:10:42.485008 5871 consensus_queue.cc:579] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Leader has been unable to successfully communicate with peer 938a24b0e6a040b7966225af26558ff8 for more than 300 seconds (300.214s)
I20240501 14:10:42.485586 5384 consensus_queue.cc:237] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: NON_VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { promote: true } }
W20240501 14:10:42.487857 5191 consensus_peers.cc:462] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:42.489223 5320 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377): Couldn't send request to peer e9c60449cf3c46f8aca4ab6c535302a4. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:42.491034 7024 raft_consensus.cc:1261] T f4d2b8c005b64bbb8e6621da068d15e5 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
W20240501 14:10:42.491243 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:42.491562 5989 consensus_queue.cc:1035] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:10:42.494112 5988 raft_consensus.cc:2932] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: NON_VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { promote: true } } }
I20240501 14:10:42.494724 7024 raft_consensus.cc:2932] T f4d2b8c005b64bbb8e6621da068d15e5 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, NON_VOTER e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130) added. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: NON_VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { promote: true } } }
I20240501 14:10:42.495901 5254 consensus_queue.cc:237] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } }
I20240501 14:10:42.497628 4839 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet f4d2b8c005b64bbb8e6621da068d15e5 with cas_config_opid_index 11: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:10:42.499402 4850 catalog_manager.cc:5484] T f4d2b8c005b64bbb8e6621da068d15e5 P 19a6d2d8a08142d5bffcde3a37234692 reported cstate change: config changed from index 11 to 12, NON_VOTER e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130) added. New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: NON_VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { promote: true } } }
I20240501 14:10:42.499861 7024 raft_consensus.cc:1261] T 404e2851055541f0be1bb0971a7cc72a P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:10:42.500304 5991 consensus_queue.cc:1035] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:10:42.500505 5989 consensus_queue.cc:579] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf [LEADER]: Leader has been unable to successfully communicate with peer 4360b59e3a154688a79ca5f9c0a37bc9 for more than 300 seconds (300.297s)
I20240501 14:10:42.500921 5125 raft_consensus.cc:1261] T 404e2851055541f0be1bb0971a7cc72a P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:10:42.501741 5991 consensus_queue.cc:1035] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:10:42.504638 5991 raft_consensus.cc:2932] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:10:42.505591 7024 raft_consensus.cc:2932] T 404e2851055541f0be1bb0971a7cc72a P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:10:42.507133 5384 consensus_queue.cc:237] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } }
I20240501 14:10:42.509183 4838 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 404e2851055541f0be1bb0971a7cc72a with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:10:42.510524 4852 catalog_manager.cc:5484] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c reported cstate change: config changed from index 7 to 8, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) evicted. New cstate: current_term: 1 leader_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 14:10:42.511883 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:42.512459 5322 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 71bdb382b08341218c8bf2459fe65fc0. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:42.512609 5125 raft_consensus.cc:2932] T 404e2851055541f0be1bb0971a7cc72a P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:10:42.516711 4993 raft_consensus.cc:1261] T 71bdb382b08341218c8bf2459fe65fc0 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:10:42.516649 5254 consensus_queue.cc:237] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: NON_VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { promote: true } }
I20240501 14:10:42.517227 5988 consensus_queue.cc:1035] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:10:42.518882 5125 raft_consensus.cc:1261] T 404e2851055541f0be1bb0971a7cc72a P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
W20240501 14:10:42.518910 5190 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129:44033): Couldn't send request to peer 04088ee3bea440a7bd5ff772152b9410. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 404e2851055541f0be1bb0971a7cc72a. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:42.519436 5991 consensus_queue.cc:1035] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:10:42.519666 7025 raft_consensus.cc:1261] T 404e2851055541f0be1bb0971a7cc72a P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:10:42.520910 5991 consensus_queue.cc:1035] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:10:42.521927 5727 raft_consensus.cc:2932] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:10:42.522634 5990 raft_consensus.cc:2932] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: NON_VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { promote: true } } }
I20240501 14:10:42.522682 4993 raft_consensus.cc:2932] T 71bdb382b08341218c8bf2459fe65fc0 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:10:42.524821 7024 raft_consensus.cc:2932] T 404e2851055541f0be1bb0971a7cc72a P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: NON_VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { promote: true } } }
I20240501 14:10:42.526203 4839 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 71bdb382b08341218c8bf2459fe65fc0 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:10:42.526768 4838 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 404e2851055541f0be1bb0971a7cc72a with cas_config_opid_index 8: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:10:42.528787 5125 raft_consensus.cc:2932] T 404e2851055541f0be1bb0971a7cc72a P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: NON_VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { promote: true } } }
I20240501 14:10:42.530143 4850 catalog_manager.cc:5484] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c reported cstate change: config changed from index 8 to 9, NON_VOTER 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129) added. New cstate: current_term: 1 leader_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: NON_VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:10:42.531381 4852 catalog_manager.cc:5484] T 71bdb382b08341218c8bf2459fe65fc0 P 04088ee3bea440a7bd5ff772152b9410 reported cstate change: config changed from index 6 to 7, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:10:42.545235 5794 consensus_queue.cc:579] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Leader has been unable to successfully communicate with peer 4360b59e3a154688a79ca5f9c0a37bc9 for more than 300 seconds (300.339s)
I20240501 14:10:42.548875 5125 consensus_queue.cc:237] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: NON_VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { promote: true } }
I20240501 14:10:42.551462 5254 raft_consensus.cc:1261] T 88ed5438fc934adc9fb98c6cb4476598 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:10:42.552093 6002 consensus_queue.cc:1035] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 14:10:42.552371 5060 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129:44033): Couldn't send request to peer 04088ee3bea440a7bd5ff772152b9410. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:42.552889 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:42.554914 5794 raft_consensus.cc:2932] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: NON_VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { promote: true } } }
I20240501 14:10:42.555917 5254 raft_consensus.cc:2932] T 88ed5438fc934adc9fb98c6cb4476598 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: NON_VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { promote: true } } }
I20240501 14:10:42.558537 4839 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 88ed5438fc934adc9fb98c6cb4476598 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:10:42.560621 4852 catalog_manager.cc:5484] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 reported cstate change: config changed from index 6 to 7, NON_VOTER 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129) added. New cstate: current_term: 1 leader_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: NON_VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:10:42.614915 5962 ts_tablet_manager.cc:902] T 99d958617b7e4ef385b94f4c73f4dd84 P 1c3fdc5272484ac09ef3e5e36a08f27c: Initiating tablet copy from peer 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129:44033)
I20240501 14:10:42.622767 5962 tablet_copy_client.cc:308] T 99d958617b7e4ef385b94f4c73f4dd84 P 1c3fdc5272484ac09ef3e5e36a08f27c: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.129:44033
I20240501 14:10:42.623507 5015 tablet_copy_service.cc:140] P 04088ee3bea440a7bd5ff772152b9410: Received BeginTabletCopySession request for tablet 99d958617b7e4ef385b94f4c73f4dd84 from peer 1c3fdc5272484ac09ef3e5e36a08f27c ({username='slave'} at 127.29.124.131:58573)
I20240501 14:10:42.623665 5015 tablet_copy_service.cc:161] P 04088ee3bea440a7bd5ff772152b9410: Beginning new tablet copy session on tablet 99d958617b7e4ef385b94f4c73f4dd84 from peer 1c3fdc5272484ac09ef3e5e36a08f27c at {username='slave'} at 127.29.124.131:58573: session id = 1c3fdc5272484ac09ef3e5e36a08f27c-99d958617b7e4ef385b94f4c73f4dd84
I20240501 14:10:42.626235 5015 tablet_copy_source_session.cc:205] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 14:10:42.627468 5962 ts_tablet_manager.cc:1875] T 99d958617b7e4ef385b94f4c73f4dd84 P 1c3fdc5272484ac09ef3e5e36a08f27c: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 14:10:42.630111 5991 consensus_queue.cc:579] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Leader has been unable to successfully communicate with peer 4360b59e3a154688a79ca5f9c0a37bc9 for more than 300 seconds (300.605s)
I20240501 14:10:42.634785 5254 consensus_queue.cc:237] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } }
I20240501 14:10:42.635110 5962 ts_tablet_manager.cc:1888] T 99d958617b7e4ef385b94f4c73f4dd84 P 1c3fdc5272484ac09ef3e5e36a08f27c: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240501 14:10:42.635493 5962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99d958617b7e4ef385b94f4c73f4dd84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:42.639257 5962 tablet_copy_client.cc:783] T 99d958617b7e4ef385b94f4c73f4dd84 P 1c3fdc5272484ac09ef3e5e36a08f27c: tablet copy: Starting download of 5 data blocks...
W20240501 14:10:42.648536 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ec1e81011c6a46bc8a39aecf3a5e71e4. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:42.648722 4993 raft_consensus.cc:1261] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 14:10:42.649017 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:42.649446 5991 consensus_queue.cc:1035] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240501 14:10:42.654266 5962 tablet_copy_client.cc:647] T 99d958617b7e4ef385b94f4c73f4dd84 P 1c3fdc5272484ac09ef3e5e36a08f27c: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:42.655959 5962 tablet_copy_client.cc:523] T 99d958617b7e4ef385b94f4c73f4dd84 P 1c3fdc5272484ac09ef3e5e36a08f27c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:42.659658 5962 tablet_bootstrap.cc:492] T 99d958617b7e4ef385b94f4c73f4dd84 P 1c3fdc5272484ac09ef3e5e36a08f27c: Bootstrap starting.
I20240501 14:10:42.663765 5871 raft_consensus.cc:2932] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } }
I20240501 14:10:42.667372 5254 raft_consensus.cc:1203] T 9d32e0707d244f2c9bbcb57f62b09f31 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240501 14:10:42.669740 4993 raft_consensus.cc:2932] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } } }
I20240501 14:10:42.671275 4838 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ec1e81011c6a46bc8a39aecf3a5e71e4 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:10:42.672634 4853 catalog_manager.cc:5484] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c reported cstate change: config changed from index 6 to 7, NON_VOTER 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) added. New cstate: current_term: 1 leader_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240501 14:10:42.681571 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 14:10:42.682757 5962 tablet_bootstrap.cc:492] T 99d958617b7e4ef385b94f4c73f4dd84 P 1c3fdc5272484ac09ef3e5e36a08f27c: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:42.683136 5962 tablet_bootstrap.cc:492] T 99d958617b7e4ef385b94f4c73f4dd84 P 1c3fdc5272484ac09ef3e5e36a08f27c: Bootstrap complete.
I20240501 14:10:42.683348 5962 ts_tablet_manager.cc:1367] T 99d958617b7e4ef385b94f4c73f4dd84 P 1c3fdc5272484ac09ef3e5e36a08f27c: Time spent bootstrapping tablet: real 0.024s user 0.019s sys 0.000s
I20240501 14:10:42.683934 5962 raft_consensus.cc:348] T 99d958617b7e4ef385b94f4c73f4dd84 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: NON_VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { promote: true } }
I20240501 14:10:42.684195 5962 raft_consensus.cc:729] T 99d958617b7e4ef385b94f4c73f4dd84 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1c3fdc5272484ac09ef3e5e36a08f27c, State: Initialized, Role: LEARNER
I20240501 14:10:42.684404 5962 consensus_queue.cc:260] T 99d958617b7e4ef385b94f4c73f4dd84 P 1c3fdc5272484ac09ef3e5e36a08f27c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: NON_VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { promote: true } }
I20240501 14:10:42.684922 5962 ts_tablet_manager.cc:1398] T 99d958617b7e4ef385b94f4c73f4dd84 P 1c3fdc5272484ac09ef3e5e36a08f27c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:10:42.685736 5015 tablet_copy_service.cc:342] P 04088ee3bea440a7bd5ff772152b9410: Request end of tablet copy session 1c3fdc5272484ac09ef3e5e36a08f27c-99d958617b7e4ef385b94f4c73f4dd84 received from {username='slave'} at 127.29.124.131:58573
I20240501 14:10:42.685863 5015 tablet_copy_service.cc:433] P 04088ee3bea440a7bd5ff772152b9410: ending tablet copy session 1c3fdc5272484ac09ef3e5e36a08f27c-99d958617b7e4ef385b94f4c73f4dd84 on tablet 99d958617b7e4ef385b94f4c73f4dd84 with peer 1c3fdc5272484ac09ef3e5e36a08f27c
I20240501 14:10:42.692956 5893 consensus_queue.cc:579] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Leader has been unable to successfully communicate with peer 4360b59e3a154688a79ca5f9c0a37bc9 for more than 300 seconds (300.315s)
W20240501 14:10:42.694567 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240501 14:10:42.696662 5125 consensus_queue.cc:237] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: NON_VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { promote: true } }
I20240501 14:10:42.699247 7285 raft_consensus.cc:1261] T ac4b234f794149e4b6def7a3b358942d P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240501 14:10:42.699589 5060 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer be56773105e744919601127a94e98adf (127.29.124.132:35459): Couldn't send request to peer be56773105e744919601127a94e98adf. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:42.699836 5794 consensus_queue.cc:1035] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240501 14:10:42.700656 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:42.708407 5794 raft_consensus.cc:2932] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER be56773105e744919601127a94e98adf (127.29.124.132) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: NON_VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { promote: true } } }
I20240501 14:10:42.708976 7285 raft_consensus.cc:2932] T ac4b234f794149e4b6def7a3b358942d P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER be56773105e744919601127a94e98adf (127.29.124.132) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: NON_VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { promote: true } } }
I20240501 14:10:42.712209 4839 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet ac4b234f794149e4b6def7a3b358942d with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:10:42.715833 4852 catalog_manager.cc:5484] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 reported cstate change: config changed from index 9 to 10, NON_VOTER be56773105e744919601127a94e98adf (127.29.124.132) added. New cstate: current_term: 1 leader_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: NON_VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:10:42.732218 5871 consensus_queue.cc:579] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Leader has been unable to successfully communicate with peer 4360b59e3a154688a79ca5f9c0a37bc9 for more than 300 seconds (300.434s)
I20240501 14:10:42.734545 5302 maintenance_manager.cc:419] P 1c3fdc5272484ac09ef3e5e36a08f27c: Scheduling UndoDeltaBlockGCOp(99d958617b7e4ef385b94f4c73f4dd84): 4834 bytes on disk
I20240501 14:10:42.735085 5193 maintenance_manager.cc:643] P 1c3fdc5272484ac09ef3e5e36a08f27c: UndoDeltaBlockGCOp(99d958617b7e4ef385b94f4c73f4dd84) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
I20240501 14:10:42.735996 5254 consensus_queue.cc:237] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } }
I20240501 14:10:42.738092 4993 raft_consensus.cc:1261] T 9fd94635797d447cbdc9181ee91f465b P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240501 14:10:42.738291 5192 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9fd94635797d447cbdc9181ee91f465b. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:42.738643 5990 consensus_queue.cc:1035] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 14:10:42.738888 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:42.741673 5871 raft_consensus.cc:2932] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:10:42.742412 4993 raft_consensus.cc:2932] T 9fd94635797d447cbdc9181ee91f465b P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:10:42.744846 4838 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9fd94635797d447cbdc9181ee91f465b with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:10:42.746771 4853 catalog_manager.cc:5484] T 9fd94635797d447cbdc9181ee91f465b P 04088ee3bea440a7bd5ff772152b9410 reported cstate change: config changed from index 6 to 7, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New cstate: current_term: 1 leader_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:10:42.797546 6055 ts_tablet_manager.cc:902] T ba6cba753e934e719c730593c0706346 P 19a6d2d8a08142d5bffcde3a37234692: Initiating tablet copy from peer 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129:44033)
I20240501 14:10:42.798676 6055 tablet_copy_client.cc:308] T ba6cba753e934e719c730593c0706346 P 19a6d2d8a08142d5bffcde3a37234692: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.129:44033
I20240501 14:10:42.806253 5015 tablet_copy_service.cc:140] P 04088ee3bea440a7bd5ff772152b9410: Received BeginTabletCopySession request for tablet ba6cba753e934e719c730593c0706346 from peer 19a6d2d8a08142d5bffcde3a37234692 ({username='slave'} at 127.29.124.134:46389)
I20240501 14:10:42.806414 5015 tablet_copy_service.cc:161] P 04088ee3bea440a7bd5ff772152b9410: Beginning new tablet copy session on tablet ba6cba753e934e719c730593c0706346 from peer 19a6d2d8a08142d5bffcde3a37234692 at {username='slave'} at 127.29.124.134:46389: session id = 19a6d2d8a08142d5bffcde3a37234692-ba6cba753e934e719c730593c0706346
I20240501 14:10:42.809183 5015 tablet_copy_source_session.cc:205] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 14:10:42.810812 6055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba6cba753e934e719c730593c0706346. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:42.815106 6055 tablet_copy_client.cc:783] T ba6cba753e934e719c730593c0706346 P 19a6d2d8a08142d5bffcde3a37234692: tablet copy: Starting download of 5 data blocks...
I20240501 14:10:42.821097 6055 tablet_copy_client.cc:647] T ba6cba753e934e719c730593c0706346 P 19a6d2d8a08142d5bffcde3a37234692: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:42.822983 6055 tablet_copy_client.cc:523] T ba6cba753e934e719c730593c0706346 P 19a6d2d8a08142d5bffcde3a37234692: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:42.826174 6055 tablet_bootstrap.cc:492] T ba6cba753e934e719c730593c0706346 P 19a6d2d8a08142d5bffcde3a37234692: Bootstrap starting.
I20240501 14:10:42.841403 6055 tablet_bootstrap.cc:492] T ba6cba753e934e719c730593c0706346 P 19a6d2d8a08142d5bffcde3a37234692: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:42.841750 6055 tablet_bootstrap.cc:492] T ba6cba753e934e719c730593c0706346 P 19a6d2d8a08142d5bffcde3a37234692: Bootstrap complete.
I20240501 14:10:42.842000 6055 ts_tablet_manager.cc:1367] T ba6cba753e934e719c730593c0706346 P 19a6d2d8a08142d5bffcde3a37234692: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.005s
I20240501 14:10:42.842523 6055 raft_consensus.cc:348] T ba6cba753e934e719c730593c0706346 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } }
I20240501 14:10:42.842716 6055 raft_consensus.cc:729] T ba6cba753e934e719c730593c0706346 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 19a6d2d8a08142d5bffcde3a37234692, State: Initialized, Role: LEARNER
I20240501 14:10:42.842888 6055 consensus_queue.cc:260] T ba6cba753e934e719c730593c0706346 P 19a6d2d8a08142d5bffcde3a37234692 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } }
I20240501 14:10:42.843340 6055 ts_tablet_manager.cc:1398] T ba6cba753e934e719c730593c0706346 P 19a6d2d8a08142d5bffcde3a37234692: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240501 14:10:42.844092 5015 tablet_copy_service.cc:342] P 04088ee3bea440a7bd5ff772152b9410: Request end of tablet copy session 19a6d2d8a08142d5bffcde3a37234692-ba6cba753e934e719c730593c0706346 received from {username='slave'} at 127.29.124.134:46389
I20240501 14:10:42.844208 5015 tablet_copy_service.cc:433] P 04088ee3bea440a7bd5ff772152b9410: ending tablet copy session 19a6d2d8a08142d5bffcde3a37234692-ba6cba753e934e719c730593c0706346 on tablet ba6cba753e934e719c730593c0706346 with peer 19a6d2d8a08142d5bffcde3a37234692
W20240501 14:10:42.856218 4840 catalog_manager.cc:4573] TS 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9d32e0707d244f2c9bbcb57f62b09f31: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
I20240501 14:10:42.900701 5989 consensus_queue.cc:579] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [LEADER]: Leader has been unable to successfully communicate with peer 938a24b0e6a040b7966225af26558ff8 for more than 300 seconds (300.211s)
W20240501 14:10:42.902566 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240501 14:10:42.904717 5384 consensus_queue.cc:237] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } }
I20240501 14:10:42.905800 5989 consensus_queue.cc:579] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [LEADER]: Leader has been unable to successfully communicate with peer 938a24b0e6a040b7966225af26558ff8 for more than 300 seconds (300.155s)
I20240501 14:10:42.907058 7285 raft_consensus.cc:1261] T 61a274b24f53458f88c2cde4f8dbe4f4 P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:10:42.907694 5727 consensus_queue.cc:1035] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:10:42.907953 7024 raft_consensus.cc:1261] T 61a274b24f53458f88c2cde4f8dbe4f4 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:10:42.908767 5727 consensus_queue.cc:1035] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:10:42.910135 5988 raft_consensus.cc:2932] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:10:42.910778 7285 raft_consensus.cc:2932] T 61a274b24f53458f88c2cde4f8dbe4f4 P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:10:42.911820 5384 consensus_queue.cc:237] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } }
I20240501 14:10:42.913969 4839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 61a274b24f53458f88c2cde4f8dbe4f4 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:10:42.914559 7024 raft_consensus.cc:2932] T 61a274b24f53458f88c2cde4f8dbe4f4 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:10:42.916754 4852 catalog_manager.cc:5484] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf reported cstate change: config changed from index 11 to 12, VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) evicted. New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:10:42.919785 7285 raft_consensus.cc:1261] T e7d5f02ccba34254a4d20951878b1214 P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240501 14:10:42.919888 7024 raft_consensus.cc:1261] T e7d5f02ccba34254a4d20951878b1214 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
I20240501 14:10:42.920557 6000 consensus_queue.cc:1035] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 14:10:42.920975 6000 consensus_queue.cc:1035] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 14:10:42.922075 6077 ts_tablet_manager.cc:902] T f4d2b8c005b64bbb8e6621da068d15e5 P e9c60449cf3c46f8aca4ab6c535302a4: Initiating tablet copy from peer be56773105e744919601127a94e98adf (127.29.124.132:35459)
I20240501 14:10:42.924286 5988 raft_consensus.cc:2932] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [term 1 LEADER]: Committing config change with OpId 1.14: config changed from index 13 to 14, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:10:42.924593 5384 consensus_queue.cc:237] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: NON_VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { promote: true } }
I20240501 14:10:42.926024 7024 raft_consensus.cc:2932] T e7d5f02ccba34254a4d20951878b1214 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
W20240501 14:10:42.927006 5320 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129:44033): Couldn't send request to peer 04088ee3bea440a7bd5ff772152b9410. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 61a274b24f53458f88c2cde4f8dbe4f4. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:42.927017 7025 raft_consensus.cc:1261] T 61a274b24f53458f88c2cde4f8dbe4f4 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:10:42.928267 7071 maintenance_manager.cc:419] P 19a6d2d8a08142d5bffcde3a37234692: Scheduling UndoDeltaBlockGCOp(ba6cba753e934e719c730593c0706346): 4921 bytes on disk
I20240501 14:10:42.928916 6962 maintenance_manager.cc:643] P 19a6d2d8a08142d5bffcde3a37234692: UndoDeltaBlockGCOp(ba6cba753e934e719c730593c0706346) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4}
I20240501 14:10:42.930490 4839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e7d5f02ccba34254a4d20951878b1214 with cas_config_opid_index 13: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:10:42.930613 5989 consensus_queue.cc:1035] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:10:42.931113 7286 raft_consensus.cc:1261] T 61a274b24f53458f88c2cde4f8dbe4f4 P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:10:42.931000 7285 raft_consensus.cc:2932] T e7d5f02ccba34254a4d20951878b1214 P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Committing config change with OpId 1.14: config changed from index 13 to 14, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:10:42.931649 5989 consensus_queue.cc:1035] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20240501 14:10:42.932274 4847 catalog_manager.cc:5484] T e7d5f02ccba34254a4d20951878b1214 P 19a6d2d8a08142d5bffcde3a37234692 reported cstate change: config changed from index 13 to 14, NON_VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) evicted. New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:10:42.934648 6000 raft_consensus.cc:2932] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [term 1 LEADER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: NON_VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { promote: true } } }
I20240501 14:10:42.935271 7286 raft_consensus.cc:2932] T 61a274b24f53458f88c2cde4f8dbe4f4 P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: NON_VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { promote: true } } }
W20240501 14:10:42.937153 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 62 similar messages]
I20240501 14:10:42.937821 4839 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 61a274b24f53458f88c2cde4f8dbe4f4 with cas_config_opid_index 12: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:10:42.939219 7024 raft_consensus.cc:2932] T 61a274b24f53458f88c2cde4f8dbe4f4 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Committing config change with OpId 1.13: config changed from index 12 to 13, NON_VOTER 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129) added. New config: { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: NON_VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { promote: true } } }
I20240501 14:10:42.942770 4852 catalog_manager.cc:5484] T 61a274b24f53458f88c2cde4f8dbe4f4 P 6dac90f01e744f9c8701a404440a4c77 reported cstate change: config changed from index 12 to 13, NON_VOTER 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129) added. New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: NON_VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { promote: true } } }
I20240501 14:10:42.943684 5384 consensus_queue.cc:237] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 1.14, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: NON_VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { promote: true } }
I20240501 14:10:42.945648 7286 raft_consensus.cc:1261] T e7d5f02ccba34254a4d20951878b1214 P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20240501 14:10:42.946228 5727 consensus_queue.cc:1035] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
W20240501 14:10:42.946785 5320 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129:44033): Couldn't send request to peer 04088ee3bea440a7bd5ff772152b9410. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:42.949249 5988 raft_consensus.cc:2932] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [term 1 LEADER]: Committing config change with OpId 1.15: config changed from index 14 to 15, NON_VOTER 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: NON_VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { promote: true } } }
I20240501 14:10:42.951752 4839 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet e7d5f02ccba34254a4d20951878b1214 with cas_config_opid_index 14: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:10:42.948048 7024 raft_consensus.cc:1261] T e7d5f02ccba34254a4d20951878b1214 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 14. Preceding OpId from leader: term: 1 index: 15. (index mismatch)
I20240501 14:10:42.952888 7286 raft_consensus.cc:2932] T e7d5f02ccba34254a4d20951878b1214 P 6dac90f01e744f9c8701a404440a4c77 [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, NON_VOTER 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: NON_VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { promote: true } } }
I20240501 14:10:42.953594 5989 consensus_queue.cc:1035] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
I20240501 14:10:42.953440 4852 catalog_manager.cc:5484] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf reported cstate change: config changed from index 14 to 15, NON_VOTER 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129) added. New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: NON_VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:10:42.956916 7024 raft_consensus.cc:2932] T e7d5f02ccba34254a4d20951878b1214 P 19a6d2d8a08142d5bffcde3a37234692 [term 1 FOLLOWER]: Committing config change with OpId 1.15: config changed from index 14 to 15, NON_VOTER 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: NON_VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { promote: true } } }
I20240501 14:10:42.961975 6077 tablet_copy_client.cc:308] T f4d2b8c005b64bbb8e6621da068d15e5 P e9c60449cf3c46f8aca4ab6c535302a4: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.132:35459
I20240501 14:10:42.962044 6081 ts_tablet_manager.cc:902] T 88ed5438fc934adc9fb98c6cb4476598 P 04088ee3bea440a7bd5ff772152b9410: Initiating tablet copy from peer e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377)
I20240501 14:10:42.962769 5406 tablet_copy_service.cc:140] P be56773105e744919601127a94e98adf: Received BeginTabletCopySession request for tablet f4d2b8c005b64bbb8e6621da068d15e5 from peer e9c60449cf3c46f8aca4ab6c535302a4 ({username='slave'} at 127.29.124.130:56443)
I20240501 14:10:42.962925 5406 tablet_copy_service.cc:161] P be56773105e744919601127a94e98adf: Beginning new tablet copy session on tablet f4d2b8c005b64bbb8e6621da068d15e5 from peer e9c60449cf3c46f8aca4ab6c535302a4 at {username='slave'} at 127.29.124.130:56443: session id = e9c60449cf3c46f8aca4ab6c535302a4-f4d2b8c005b64bbb8e6621da068d15e5
I20240501 14:10:42.963655 6081 tablet_copy_client.cc:308] T 88ed5438fc934adc9fb98c6cb4476598 P 04088ee3bea440a7bd5ff772152b9410: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.130:34377
I20240501 14:10:42.965843 5406 tablet_copy_source_session.cc:205] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 14:10:42.967356 6077 ts_tablet_manager.cc:1875] T f4d2b8c005b64bbb8e6621da068d15e5 P e9c60449cf3c46f8aca4ab6c535302a4: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 14:10:42.970880 5146 tablet_copy_service.cc:140] P e9c60449cf3c46f8aca4ab6c535302a4: Received BeginTabletCopySession request for tablet 88ed5438fc934adc9fb98c6cb4476598 from peer 04088ee3bea440a7bd5ff772152b9410 ({username='slave'} at 127.29.124.129:55297)
I20240501 14:10:42.971024 5146 tablet_copy_service.cc:161] P e9c60449cf3c46f8aca4ab6c535302a4: Beginning new tablet copy session on tablet 88ed5438fc934adc9fb98c6cb4476598 from peer 04088ee3bea440a7bd5ff772152b9410 at {username='slave'} at 127.29.124.129:55297: session id = 04088ee3bea440a7bd5ff772152b9410-88ed5438fc934adc9fb98c6cb4476598
I20240501 14:10:42.973913 5146 tablet_copy_source_session.cc:205] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 14:10:42.974341 6077 ts_tablet_manager.cc:1888] T f4d2b8c005b64bbb8e6621da068d15e5 P e9c60449cf3c46f8aca4ab6c535302a4: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240501 14:10:42.974637 6077 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f4d2b8c005b64bbb8e6621da068d15e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:42.975534 6081 ts_tablet_manager.cc:1875] T 88ed5438fc934adc9fb98c6cb4476598 P 04088ee3bea440a7bd5ff772152b9410: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 14:10:42.978747 6077 tablet_copy_client.cc:783] T f4d2b8c005b64bbb8e6621da068d15e5 P e9c60449cf3c46f8aca4ab6c535302a4: tablet copy: Starting download of 5 data blocks...
I20240501 14:10:42.979256 6091 ts_tablet_manager.cc:902] T 71bdb382b08341218c8bf2459fe65fc0 P 6dac90f01e744f9c8701a404440a4c77: Initiating tablet copy from peer be56773105e744919601127a94e98adf (127.29.124.132:35459)
F20240501 14:10:42.979723 6091 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-1a7bbfc15a76450bbc9197037e856e08.queue_time_us, tablet-download-pool-1a7bbfc15a76450bbc9197037e856e08.queue_time_us) (tablet-download-pool-1a7bbfc15a76450bbc9197037e856e08.run_cpu_time_us, tablet-download-pool-1a7bbfc15a76450bbc9197037e856e08.run_cpu_time_us) (tablet-download-pool-1a7bbfc15a76450bbc9197037e856e08.run_wall_time_us, tablet-download-pool-1a7bbfc15a76450bbc9197037e856e08.run_wall_time_us) (tablet-download-pool-23ca2bf1e59f44fcac8a48aff8d056b1.queue_time_us, tablet-download-pool-23ca2bf1e59f44fcac8a48aff8d056b1.queue_time_us) (tablet-download-pool-23ca2bf1e59f44fcac8a48aff8d056b1.run_cpu_time_us, tablet-download-pool-23ca2bf1e59f44fcac8a48aff8d056b1.run_cpu_time_us) (tablet-download-pool-23ca2bf1e59f44fcac8a48aff8d056b1.run_wall_time_us, tablet-download-pool-23ca2bf1e59f44fcac8a48aff8d056b1.run_wall_time_us) (tablet-download-pool-2c39b53867e047faaf24cc6f87a8100c.queue_time_us, tablet-download-pool-2c39b53867e047faaf24cc6f87a8100c.queue_time_us) (tablet-download-pool-2c39b53867e047faaf24cc6f87a8100c.run_cpu_time_us, tablet-download-pool-2c39b53867e047faaf24cc6f87a8100c.run_cpu_time_us) (tablet-download-pool-2c39b53867e047faaf24cc6f87a8100c.run_wall_time_us, tablet-download-pool-2c39b53867e047faaf24cc6f87a8100c.run_wall_time_us) (tablet-download-pool-420642af520d45ffbc7f2d35f93e75a7.queue_time_us, tablet-download-pool-420642af520d45ffbc7f2d35f93e75a7.queue_time_us) (tablet-download-pool-420642af520d45ffbc7f2d35f93e75a7.run_cpu_time_us, tablet-download-pool-420642af520d45ffbc7f2d35f93e75a7.run_cpu_time_us) (tablet-download-pool-420642af520d45ffbc7f2d35f93e75a7.run_wall_time_us, tablet-download-pool-420642af520d45ffbc7f2d35f93e75a7.run_wall_time_us) (tablet-download-pool-43b46fcf827c4275a47291afb107c290.queue_time_us, tablet-download-pool-43b46fcf827c4275a47291afb107c290.queue_time_us) (tablet-download-pool-43b46fcf827c4275a47291afb107c290.run_cpu_time_us, tablet-download-pool-43b46fcf827c4275a47291afb107c290.run_cpu_time_us) (tablet-download-pool-43b46fcf827c4275a47291afb107c290.run_wall_time_us, tablet-download-pool-43b46fcf827c4275a47291afb107c290.run_wall_time_us) (tablet-download-pool-61a274b24f53458f88c2cde4f8dbe4f4.queue_time_us, tablet-download-pool-61a274b24f53458f88c2cde4f8dbe4f4.queue_time_us) (tablet-download-pool-61a274b24f53458f88c2cde4f8dbe4f4.run_cpu_time_us, tablet-download-pool-61a274b24f53458f88c2cde4f8dbe4f4.run_cpu_time_us) (tablet-download-pool-61a274b24f53458f88c2cde4f8dbe4f4.run_wall_time_us, tablet-download-pool-61a274b24f53458f88c2cde4f8dbe4f4.run_wall_time_us) (tablet-download-pool-71660575d94748d8a9329f909a42a0cd.queue_time_us, tablet-download-pool-71660575d94748d8a9329f909a42a0cd.queue_time_us) (tablet-download-pool-71660575d94748d8a9329f909a42a0cd.run_cpu_time_us, tablet-download-pool-71660575d94748d8a9329f909a42a0cd.run_cpu_time_us) (tablet-download-pool-71660575d94748d8a9329f909a42a0cd.run_wall_time_us, tablet-download-pool-71660575d94748d8a9329f909a42a0cd.run_wall_time_us) (tablet-download-pool-71bdb382b08341218c8bf2459fe65fc0.queue_time_us, tablet-download-pool-71bdb382b08341218c8bf2459fe65fc0.queue_time_us) (tablet-download-pool-9afe3b20a43d4b5bb800af39f945948d.queue_time_us, tablet-download-pool-9afe3b20a43d4b5bb800af39f945948d.queue_time_us) (tablet-download-pool-9afe3b20a43d4b5bb800af39f945948d.run_cpu_time_us, tablet-download-pool-9afe3b20a43d4b5bb800af39f945948d.run_cpu_time_us) (tablet-download-pool-9afe3b20a43d4b5bb800af39f945948d.run_wall_time_us, tablet-download-pool-9afe3b20a43d4b5bb800af39f945948d.run_wall_time_us) (tablet-download-pool-9b88c43cb8704583a3e65c5c35c205f6.queue_time_us, tablet-download-pool-9b88c43cb8704583a3e65c5c35c205f6.queue_time_us) (tablet-download-pool-9b88c43cb8704583a3e65c5c35c205f6.run_cpu_time_us, tablet-download-pool-9b88c43cb8704583a3e65c5c35c205f6.run_cpu_time_us) (tablet-download-pool-9b88c43cb8704583a3e65c5c35c205f6.run_wall_time_us, tablet-download-pool-9b88c43cb8704583a3e65c5c35c205f6.run_wall_time_us) (tablet-download-pool-9d32e0707d244f2c9bbcb57f62b09f31.queue_time_us, tablet-download-pool-9d32e0707d244f2c9bbcb57f62b09f31.queue_time_us) (tablet-download-pool-9d32e0707d244f2c9bbcb57f62b09f31.run_cpu_time_us, tablet-download-pool-9d32e0707d244f2c9bbcb57f62b09f31.run_cpu_time_us) (tablet-download-pool-9d32e0707d244f2c9bbcb57f62b09f31.run_wall_time_us, tablet-download-pool-9d32e0707d244f2c9bbcb57f62b09f31.run_wall_time_us) (tablet-download-pool-ac4b234f794149e4b6def7a3b358942d.queue_time_us, tablet-download-pool-ac4b234f794149e4b6def7a3b358942d.queue_time_us) (tablet-download-pool-ac4b234f794149e4b6def7a3b358942d.run_cpu_time_us, tablet-download-pool-ac4b234f794149e4b6def7a3b358942d.run_cpu_time_us) (tablet-download-pool-ac4b234f794149e4b6def7a3b358942d.run_wall_time_us, tablet-download-pool-ac4b234f794149e4b6def7a3b358942d.run_wall_time_us) (tablet-download-pool-de6c7753c8564107b858876fcfa5924d.queue_time_us, tablet-download-pool-de6c7753c8564107b858876fcfa5924d.queue_time_us) (tablet-download-pool-de6c7753c8564107b858876fcfa5924d.run_cpu_time_us, tablet-download-pool-de6c7753c8564107b858876fcfa5924d.run_cpu_time_us) (tablet-download-pool-de6c7753c8564107b858876fcfa5924d.run_wall_time_us, tablet-download-pool-de6c7753c8564107b858876fcfa5924d.run_wall_time_us) (tablet-download-pool-e7d5f02ccba34254a4d20951878b1214.queue_time_us, tablet-download-pool-e7d5f02ccba34254a4d20951878b1214.queue_time_us) (tablet-download-pool-e7d5f02ccba34254a4d20951878b1214.run_cpu_time_us, tablet-download-pool-e7d5f02ccba34254a4d20951878b1214.run_cpu_time_us) (tablet-download-pool-e7d5f02ccba34254a4d20951878b1214.run_wall_time_us, tablet-download-pool-e7d5f02ccba34254a4d20951878b1214.run_wall_time_us) (tablet-download-pool-f9cb784f60774ea7b6e70ccfebf867a8.queue_time_us, tablet-download-pool-f9cb784f60774ea7b6e70ccfebf867a8.queue_time_us) (tablet-download-pool-f9cb784f60774ea7b6e70ccfebf867a8.run_cpu_time_us, tablet-download-pool-f9cb784f60774ea7b6e70ccfebf867a8.run_cpu_time_us) (tablet-download-pool-f9cb784f60774ea7b6e70ccfebf867a8.run_wall_time_us, tablet-download-pool-f9cb784f60774ea7b6e70ccfebf867a8.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7efe665f8dcd google::LogMessage::Fail() at ??:0
I20240501 14:10:42.982932 6081 ts_tablet_manager.cc:1888] T 88ed5438fc934adc9fb98c6cb4476598 P 04088ee3bea440a7bd5ff772152b9410: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240501 14:10:42.983211 6081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88ed5438fc934adc9fb98c6cb4476598. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7efe665fcb93 google::LogMessage::SendToLog() at ??:0
I20240501 14:10:42.985603 6077 tablet_copy_client.cc:647] T f4d2b8c005b64bbb8e6621da068d15e5 P e9c60449cf3c46f8aca4ab6c535302a4: tablet copy: Starting download of 1 WAL segments...
@ 0x7efe665f87cc google::LogMessage::Flush() at ??:0
I20240501 14:10:42.987522 6081 tablet_copy_client.cc:783] T 88ed5438fc934adc9fb98c6cb4476598 P 04088ee3bea440a7bd5ff772152b9410: tablet copy: Starting download of 5 data blocks...
I20240501 14:10:42.987572 6077 tablet_copy_client.cc:523] T f4d2b8c005b64bbb8e6621da068d15e5 P e9c60449cf3c46f8aca4ab6c535302a4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
@ 0x7efe665f9f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240501 14:10:42.991990 6077 tablet_bootstrap.cc:492] T f4d2b8c005b64bbb8e6621da068d15e5 P e9c60449cf3c46f8aca4ab6c535302a4: Bootstrap starting.
@ 0x7efe69339742 kudu::TraceMetrics::InternName() at ??:0
I20240501 14:10:42.995846 6081 tablet_copy_client.cc:647] T 88ed5438fc934adc9fb98c6cb4476598 P 04088ee3bea440a7bd5ff772152b9410: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:42.998000 5254 raft_consensus.cc:1203] T 99d958617b7e4ef385b94f4c73f4dd84 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:10:42.998667 6081 tablet_copy_client.cc:523] T 88ed5438fc934adc9fb98c6cb4476598 P 04088ee3bea440a7bd5ff772152b9410: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:43.003073 6081 tablet_bootstrap.cc:492] T 88ed5438fc934adc9fb98c6cb4476598 P 04088ee3bea440a7bd5ff772152b9410: Bootstrap starting.
@ 0x7efe69305f8b kudu::ThreadPool::ThreadPool() at ??:0
W20240501 14:10:43.017040 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
@ 0x7efe692ffcd6 kudu::ThreadPoolBuilder::Build() at ??:0
@ 0x7efe787528de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240501 14:10:43.024650 6077 tablet_bootstrap.cc:492] T f4d2b8c005b64bbb8e6621da068d15e5 P e9c60449cf3c46f8aca4ab6c535302a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=12 overwritten=0 applied=12 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:43.024930 6077 tablet_bootstrap.cc:492] T f4d2b8c005b64bbb8e6621da068d15e5 P e9c60449cf3c46f8aca4ab6c535302a4: Bootstrap complete.
I20240501 14:10:43.025133 6077 ts_tablet_manager.cc:1367] T f4d2b8c005b64bbb8e6621da068d15e5 P e9c60449cf3c46f8aca4ab6c535302a4: Time spent bootstrapping tablet: real 0.033s user 0.011s sys 0.009s
@ 0x7efe7876adaf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240501 14:10:43.028710 6077 raft_consensus.cc:348] T f4d2b8c005b64bbb8e6621da068d15e5 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: NON_VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { promote: true } }
I20240501 14:10:43.028973 6077 raft_consensus.cc:729] T f4d2b8c005b64bbb8e6621da068d15e5 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e9c60449cf3c46f8aca4ab6c535302a4, State: Initialized, Role: LEARNER
I20240501 14:10:43.029295 6077 consensus_queue.cc:260] T f4d2b8c005b64bbb8e6621da068d15e5 P e9c60449cf3c46f8aca4ab6c535302a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: NON_VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { promote: true } }
@ 0x7efe78895480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240501 14:10:43.032145 6077 ts_tablet_manager.cc:1398] T f4d2b8c005b64bbb8e6621da068d15e5 P e9c60449cf3c46f8aca4ab6c535302a4: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20240501 14:10:43.033021 5406 tablet_copy_service.cc:342] P be56773105e744919601127a94e98adf: Request end of tablet copy session e9c60449cf3c46f8aca4ab6c535302a4-f4d2b8c005b64bbb8e6621da068d15e5 received from {username='slave'} at 127.29.124.130:56443
I20240501 14:10:43.033140 5406 tablet_copy_service.cc:433] P be56773105e744919601127a94e98adf: ending tablet copy session e9c60449cf3c46f8aca4ab6c535302a4-f4d2b8c005b64bbb8e6621da068d15e5 on tablet f4d2b8c005b64bbb8e6621da068d15e5 with peer e9c60449cf3c46f8aca4ab6c535302a4
I20240501 14:10:43.034905 6081 tablet_bootstrap.cc:492] T 88ed5438fc934adc9fb98c6cb4476598 P 04088ee3bea440a7bd5ff772152b9410: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:43.035295 6081 tablet_bootstrap.cc:492] T 88ed5438fc934adc9fb98c6cb4476598 P 04088ee3bea440a7bd5ff772152b9410: Bootstrap complete.
I20240501 14:10:43.035569 6081 ts_tablet_manager.cc:1367] T 88ed5438fc934adc9fb98c6cb4476598 P 04088ee3bea440a7bd5ff772152b9410: Time spent bootstrapping tablet: real 0.033s user 0.020s sys 0.000s
@ 0x7efe788cda8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240501 14:10:43.036159 6081 raft_consensus.cc:348] T 88ed5438fc934adc9fb98c6cb4476598 P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: NON_VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { promote: true } }
I20240501 14:10:43.036346 6081 raft_consensus.cc:729] T 88ed5438fc934adc9fb98c6cb4476598 P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 04088ee3bea440a7bd5ff772152b9410, State: Initialized, Role: LEARNER
I20240501 14:10:43.036510 6081 consensus_queue.cc:260] T 88ed5438fc934adc9fb98c6cb4476598 P 04088ee3bea440a7bd5ff772152b9410 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: NON_VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { promote: true } }
I20240501 14:10:43.037204 6081 ts_tablet_manager.cc:1398] T 88ed5438fc934adc9fb98c6cb4476598 P 04088ee3bea440a7bd5ff772152b9410: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240501 14:10:43.038326 5146 tablet_copy_service.cc:342] P e9c60449cf3c46f8aca4ab6c535302a4: Request end of tablet copy session 04088ee3bea440a7bd5ff772152b9410-88ed5438fc934adc9fb98c6cb4476598 received from {username='slave'} at 127.29.124.129:55297
I20240501 14:10:43.038436 5146 tablet_copy_service.cc:433] P e9c60449cf3c46f8aca4ab6c535302a4: ending tablet copy session 04088ee3bea440a7bd5ff772152b9410-88ed5438fc934adc9fb98c6cb4476598 on tablet 88ed5438fc934adc9fb98c6cb4476598 with peer 04088ee3bea440a7bd5ff772152b9410
@ 0x7efe788b0981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7efe788b070e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7efe69309b75 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7efe6930d19c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7efe6930d019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7efe692e801e kudu::Thread::SuperviseThread() at ??:0
I20240501 14:10:43.062448 5041 maintenance_manager.cc:419] P 04088ee3bea440a7bd5ff772152b9410: Scheduling UndoDeltaBlockGCOp(88ed5438fc934adc9fb98c6cb4476598): 4581 bytes on disk
@ 0x7efe6c66d6db start_thread at ??:0
I20240501 14:10:43.063256 4932 maintenance_manager.cc:643] P 04088ee3bea440a7bd5ff772152b9410: UndoDeltaBlockGCOp(88ed5438fc934adc9fb98c6cb4476598) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
@ 0x7efe654ab71f clone at ??:0
I20240501 14:10:43.081971 6081 ts_tablet_manager.cc:902] T 404e2851055541f0be1bb0971a7cc72a P 04088ee3bea440a7bd5ff772152b9410: Initiating tablet copy from peer 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209)
I20240501 14:10:43.083721 6055 ts_tablet_manager.cc:902] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 19a6d2d8a08142d5bffcde3a37234692: Initiating tablet copy from peer 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209)
F20240501 14:10:43.084076 6055 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-1a7bbfc15a76450bbc9197037e856e08.queue_time_us, tablet-download-pool-1a7bbfc15a76450bbc9197037e856e08.queue_time_us) (tablet-download-pool-1a7bbfc15a76450bbc9197037e856e08.run_cpu_time_us, tablet-download-pool-1a7bbfc15a76450bbc9197037e856e08.run_cpu_time_us) (tablet-download-pool-1a7bbfc15a76450bbc9197037e856e08.run_wall_time_us, tablet-download-pool-1a7bbfc15a76450bbc9197037e856e08.run_wall_time_us) (tablet-download-pool-23ca2bf1e59f44fcac8a48aff8d056b1.queue_time_us, tablet-download-pool-23ca2bf1e59f44fcac8a48aff8d056b1.queue_time_us) (tablet-download-pool-23ca2bf1e59f44fcac8a48aff8d056b1.run_cpu_time_us, tablet-download-pool-23ca2bf1e59f44fcac8a48aff8d056b1.run_cpu_time_us) (tablet-download-pool-23ca2bf1e59f44fcac8a48aff8d056b1.run_wall_time_us, tablet-download-pool-23ca2bf1e59f44fcac8a48aff8d056b1.run_wall_time_us) (tablet-download-pool-2c39b53867e047faaf24cc6f87a8100c.queue_time_us, tablet-download-pool-2c39b53867e047faaf24cc6f87a8100c.queue_time_us) (tablet-download-pool-2c39b53867e047faaf24cc6f87a8100c.run_cpu_time_us, tablet-download-pool-2c39b53867e047faaf24cc6f87a8100c.run_cpu_time_us) (tablet-download-pool-2c39b53867e047faaf24cc6f87a8100c.run_wall_time_us, tablet-download-pool-2c39b53867e047faaf24cc6f87a8100c.run_wall_time_us) (tablet-download-pool-3d6859d53fd747a5a5e94722626fa522.queue_time_us, tablet-download-pool-3d6859d53fd747a5a5e94722626fa522.queue_time_us) (tablet-download-pool-3d6859d53fd747a5a5e94722626fa522.run_cpu_time_us, tablet-download-pool-3d6859d53fd747a5a5e94722626fa522.run_cpu_time_us) (tablet-download-pool-3d6859d53fd747a5a5e94722626fa522.run_wall_time_us, tablet-download-pool-3d6859d53fd747a5a5e94722626fa522.run_wall_time_us) (tablet-download-pool-404e2851055541f0be1bb0971a7cc72a.queue_time_us, tablet-download-pool-404e2851055541f0be1bb0971a7cc72a.queue_time_us) (tablet-download-pool-404e2851055541f0be1bb0971a7cc72a.run_cpu_time_us, tablet-download-pool-404e2851055541f0be1bb0971a7cc72a.run_cpu_time_us) (tablet-download-pool-404e2851055541f0be1bb0971a7cc72a.run_wall_time_us, tablet-download-pool-404e2851055541f0be1bb0971a7cc72a.run_wall_time_us) (tablet-download-pool-43b46fcf827c4275a47291afb107c290.queue_time_us, tablet-download-pool-43b46fcf827c4275a47291afb107c290.queue_time_us) (tablet-download-pool-43b46fcf827c4275a47291afb107c290.run_cpu_time_us, tablet-download-pool-43b46fcf827c4275a47291afb107c290.run_cpu_time_us) (tablet-download-pool-43b46fcf827c4275a47291afb107c290.run_wall_time_us, tablet-download-pool-43b46fcf827c4275a47291afb107c290.run_wall_time_us) (tablet-download-pool-61a274b24f53458f88c2cde4f8dbe4f4.queue_time_us, tablet-download-pool-61a274b24f53458f88c2cde4f8dbe4f4.queue_time_us) (tablet-download-pool-61a274b24f53458f88c2cde4f8dbe4f4.run_cpu_time_us, tablet-download-pool-61a274b24f53458f88c2cde4f8dbe4f4.run_cpu_time_us) (tablet-download-pool-61a274b24f53458f88c2cde4f8dbe4f4.run_wall_time_us, tablet-download-pool-61a274b24f53458f88c2cde4f8dbe4f4.run_wall_time_us) (tablet-download-pool-8cc5eec4600640f7a293877c825f8d96.queue_time_us, tablet-download-pool-8cc5eec4600640f7a293877c825f8d96.queue_time_us) (tablet-download-pool-8cc5eec4600640f7a293877c825f8d96.run_cpu_time_us, tablet-download-pool-8cc5eec4600640f7a293877c825f8d96.run_cpu_time_us) (tablet-download-pool-8cc5eec4600640f7a293877c825f8d96.run_wall_time_us, tablet-download-pool-8cc5eec4600640f7a293877c825f8d96.run_wall_time_us) (tablet-download-pool-a726769d24284fedadd8e9e63cd27d05.queue_time_us, tablet-download-pool-a726769d24284fedadd8e9e63cd27d05.queue_time_us) (tablet-download-pool-a726769d24284fedadd8e9e63cd27d05.run_cpu_time_us, tablet-download-pool-a726769d24284fedadd8e9e63cd27d05.run_cpu_time_us) (tablet-download-pool-a726769d24284fedadd8e9e63cd27d05.run_wall_time_us, tablet-download-pool-a726769d24284fedadd8e9e63cd27d05.run_wall_time_us) (tablet-download-pool-ba6cba753e934e719c730593c0706346.queue_time_us, tablet-download-pool-ba6cba753e934e719c730593c0706346.queue_time_us) (tablet-download-pool-ba6cba753e934e719c730593c0706346.run_cpu_time_us, tablet-download-pool-ba6cba753e934e719c730593c0706346.run_cpu_time_us) (tablet-download-pool-ba6cba753e934e719c730593c0706346.run_wall_time_us, tablet-download-pool-ba6cba753e934e719c730593c0706346.run_wall_time_us) (tablet-download-pool-e7d5f02ccba34254a4d20951878b1214.queue_time_us, tablet-download-pool-e7d5f02ccba34254a4d20951878b1214.queue_time_us) (tablet-download-pool-e7d5f02ccba34254a4d20951878b1214.run_cpu_time_us, tablet-download-pool-e7d5f02ccba34254a4d20951878b1214.run_cpu_time_us) (tablet-download-pool-e7d5f02ccba34254a4d20951878b1214.run_wall_time_us, tablet-download-pool-e7d5f02ccba34254a4d20951878b1214.run_wall_time_us) (tablet-download-pool-ec1e81011c6a46bc8a39aecf3a5e71e4.queue_time_us, tablet-download-pool-ec1e81011c6a46bc8a39aecf3a5e71e4.queue_time_us) (tablet-download-pool-f4d2b8c005b64bbb8e6621da068d15e5.queue_time_us, tablet-download-pool-f4d2b8c005b64bbb8e6621da068d15e5.queue_time_us) (tablet-download-pool-f4d2b8c005b64bbb8e6621da068d15e5.run_cpu_time_us, tablet-download-pool-f4d2b8c005b64bbb8e6621da068d15e5.run_cpu_time_us) (tablet-download-pool-f4d2b8c005b64bbb8e6621da068d15e5.run_wall_time_us, tablet-download-pool-f4d2b8c005b64bbb8e6621da068d15e5.run_wall_time_us) (tablet-download-pool-f9cb784f60774ea7b6e70ccfebf867a8.queue_time_us, tablet-download-pool-f9cb784f60774ea7b6e70ccfebf867a8.queue_time_us) (tablet-download-pool-f9cb784f60774ea7b6e70ccfebf867a8.run_cpu_time_us, tablet-download-pool-f9cb784f60774ea7b6e70ccfebf867a8.run_cpu_time_us) (tablet-download-pool-f9cb784f60774ea7b6e70ccfebf867a8.run_wall_time_us, tablet-download-pool-f9cb784f60774ea7b6e70ccfebf867a8.run_wall_time_us) (tablet-download-pool-fee16905387d428aa02cd2070c86c790.queue_time_us, tablet-download-pool-fee16905387d428aa02cd2070c86c790.queue_time_us) (tablet-download-pool-fee16905387d428aa02cd2070c86c790.run_cpu_time_us, tablet-download-pool-fee16905387d428aa02cd2070c86c790.run_cpu_time_us) (tablet-download-pool-fee16905387d428aa02cd2070c86c790.run_wall_time_us, tablet-download-pool-fee16905387d428aa02cd2070c86c790.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
@ 0x7f4521a7edcd google::LogMessage::Fail() at ??:0
@ 0x7f4521a82b93 google::LogMessage::SendToLog() at ??:0
@ 0x7f4521a7e7cc google::LogMessage::Flush() at ??:0
@ 0x7f4521a7ff59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f45247bf742 kudu::TraceMetrics::InternName() at ??:0
I20240501 14:10:43.108938 6081 tablet_copy_client.cc:308] T 404e2851055541f0be1bb0971a7cc72a P 04088ee3bea440a7bd5ff772152b9410: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.131:33209
@ 0x7f452478bf8b kudu::ThreadPool::ThreadPool() at ??:0
I20240501 14:10:43.109622 5276 tablet_copy_service.cc:140] P 1c3fdc5272484ac09ef3e5e36a08f27c: Received BeginTabletCopySession request for tablet 404e2851055541f0be1bb0971a7cc72a from peer 04088ee3bea440a7bd5ff772152b9410 ({username='slave'} at 127.29.124.129:42285)
I20240501 14:10:43.109778 5276 tablet_copy_service.cc:161] P 1c3fdc5272484ac09ef3e5e36a08f27c: Beginning new tablet copy session on tablet 404e2851055541f0be1bb0971a7cc72a from peer 04088ee3bea440a7bd5ff772152b9410 at {username='slave'} at 127.29.124.129:42285: session id = 04088ee3bea440a7bd5ff772152b9410-404e2851055541f0be1bb0971a7cc72a
@ 0x7f4524785cd6 kudu::ThreadPoolBuilder::Build() at ??:0
I20240501 14:10:43.113886 5276 tablet_copy_source_session.cc:205] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 14:10:43.115695 6081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 404e2851055541f0be1bb0971a7cc72a. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f4533bd88de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
@ 0x7f4533bf0daf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
W20240501 14:10:43.124778 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240501 14:10:43.126039 6125 ts_tablet_manager.cc:902] T ac4b234f794149e4b6def7a3b358942d P be56773105e744919601127a94e98adf: Initiating tablet copy from peer e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377)
I20240501 14:10:43.126886 6081 tablet_copy_client.cc:783] T 404e2851055541f0be1bb0971a7cc72a P 04088ee3bea440a7bd5ff772152b9410: tablet copy: Starting download of 5 data blocks...
@ 0x7f4533d1b480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240501 14:10:43.134449 6081 tablet_copy_client.cc:647] T 404e2851055541f0be1bb0971a7cc72a P 04088ee3bea440a7bd5ff772152b9410: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:43.137375 6081 tablet_copy_client.cc:523] T 404e2851055541f0be1bb0971a7cc72a P 04088ee3bea440a7bd5ff772152b9410: tablet copy: Tablet Copy complete. Replacing tablet superblock.
@ 0x7f4533d53a8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240501 14:10:43.141168 6081 tablet_bootstrap.cc:492] T 404e2851055541f0be1bb0971a7cc72a P 04088ee3bea440a7bd5ff772152b9410: Bootstrap starting.
@ 0x7f4533d36981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7f4533d3670e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
I20240501 14:10:43.149681 6044 raft_consensus.cc:1050] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4: attempting to promote NON_VOTER 1c3fdc5272484ac09ef3e5e36a08f27c to VOTER
I20240501 14:10:43.150579 6044 consensus_queue.cc:237] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { promote: false } }
@ 0x7f452478fb75 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f452479319c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f4524793019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
I20240501 14:10:43.166046 6125 tablet_copy_client.cc:308] T ac4b234f794149e4b6def7a3b358942d P be56773105e744919601127a94e98adf: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.130:34377
@ 0x7f452476e01e kudu::Thread::SuperviseThread() at ??:0
I20240501 14:10:43.166898 5146 tablet_copy_service.cc:140] P e9c60449cf3c46f8aca4ab6c535302a4: Received BeginTabletCopySession request for tablet ac4b234f794149e4b6def7a3b358942d from peer be56773105e744919601127a94e98adf ({username='slave'} at 127.29.124.132:43515)
I20240501 14:10:43.167068 5146 tablet_copy_service.cc:161] P e9c60449cf3c46f8aca4ab6c535302a4: Beginning new tablet copy session on tablet ac4b234f794149e4b6def7a3b358942d from peer be56773105e744919601127a94e98adf at {username='slave'} at 127.29.124.132:43515: session id = be56773105e744919601127a94e98adf-ac4b234f794149e4b6def7a3b358942d
I20240501 14:10:43.168066 5254 raft_consensus.cc:1261] T 9d32e0707d244f2c9bbcb57f62b09f31 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEARNER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:10:43.168313 5384 raft_consensus.cc:1261] T 9d32e0707d244f2c9bbcb57f62b09f31 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:10:43.168641 6044 consensus_queue.cc:1035] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:10:43.169059 5893 consensus_queue.cc:1035] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:10:43.170006 5146 tablet_copy_source_session.cc:205] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 14:10:43.172860 6125 ts_tablet_manager.cc:1875] T ac4b234f794149e4b6def7a3b358942d P be56773105e744919601127a94e98adf: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 14:10:43.181034 6125 ts_tablet_manager.cc:1888] T ac4b234f794149e4b6def7a3b358942d P be56773105e744919601127a94e98adf: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.8
I20240501 14:10:43.181320 6125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac4b234f794149e4b6def7a3b358942d. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f4527af36db start_thread at ??:0
@ 0x7f452093171f clone at ??:0
I20240501 14:10:43.185119 6094 consensus_queue.cc:579] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Leader has been unable to successfully communicate with peer 938a24b0e6a040b7966225af26558ff8 for more than 300 seconds (300.165s)
I20240501 14:10:43.185322 6125 tablet_copy_client.cc:783] T ac4b234f794149e4b6def7a3b358942d P be56773105e744919601127a94e98adf: tablet copy: Starting download of 5 data blocks...
I20240501 14:10:43.185585 6044 raft_consensus.cc:2932] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { promote: false } } }
I20240501 14:10:43.186187 5255 raft_consensus.cc:2932] T 9d32e0707d244f2c9bbcb57f62b09f31 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { promote: false } } }
W20240501 14:10:43.187512 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240501 14:10:43.191118 4993 consensus_queue.cc:237] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: NON_VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { promote: true } }
I20240501 14:10:43.191519 4853 catalog_manager.cc:5484] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 reported cstate change: config changed from index 9 to 10, 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:10:43.194640 5384 raft_consensus.cc:2932] T 9d32e0707d244f2c9bbcb57f62b09f31 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { replace: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { promote: false } } }
W20240501 14:10:43.197695 4929 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer be56773105e744919601127a94e98adf (127.29.124.132:35459): Couldn't send request to peer be56773105e744919601127a94e98adf. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 63650547ab444f45bb377bcb49f258ad. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:43.197726 5254 raft_consensus.cc:1261] T 63650547ab444f45bb377bcb49f258ad P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240501 14:10:43.198424 6094 consensus_queue.cc:1035] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240501 14:10:43.201288 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:43.205456 6081 tablet_bootstrap.cc:492] T 404e2851055541f0be1bb0971a7cc72a P 04088ee3bea440a7bd5ff772152b9410: Bootstrap replayed 1/1 log segments. Stats: ops{read=9 overwritten=0 applied=9 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:43.205837 6081 tablet_bootstrap.cc:492] T 404e2851055541f0be1bb0971a7cc72a P 04088ee3bea440a7bd5ff772152b9410: Bootstrap complete.
I20240501 14:10:43.206095 6081 ts_tablet_manager.cc:1367] T 404e2851055541f0be1bb0971a7cc72a P 04088ee3bea440a7bd5ff772152b9410: Time spent bootstrapping tablet: real 0.065s user 0.025s sys 0.000s
I20240501 14:10:43.206928 6125 tablet_copy_client.cc:647] T ac4b234f794149e4b6def7a3b358942d P be56773105e744919601127a94e98adf: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:43.206885 6081 raft_consensus.cc:348] T 404e2851055541f0be1bb0971a7cc72a P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: NON_VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { promote: true } }
I20240501 14:10:43.207135 6081 raft_consensus.cc:729] T 404e2851055541f0be1bb0971a7cc72a P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 04088ee3bea440a7bd5ff772152b9410, State: Initialized, Role: LEARNER
I20240501 14:10:43.207317 6081 consensus_queue.cc:260] T 404e2851055541f0be1bb0971a7cc72a P 04088ee3bea440a7bd5ff772152b9410 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 9, Last appended: 1.9, Last appended by leader: 9, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: NON_VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { promote: true } }
I20240501 14:10:43.215395 5172 maintenance_manager.cc:419] P e9c60449cf3c46f8aca4ab6c535302a4: Scheduling UndoDeltaBlockGCOp(f4d2b8c005b64bbb8e6621da068d15e5): 4581 bytes on disk
I20240501 14:10:43.216009 5063 maintenance_manager.cc:643] P e9c60449cf3c46f8aca4ab6c535302a4: UndoDeltaBlockGCOp(f4d2b8c005b64bbb8e6621da068d15e5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
I20240501 14:10:43.216153 6081 ts_tablet_manager.cc:1398] T 404e2851055541f0be1bb0971a7cc72a P 04088ee3bea440a7bd5ff772152b9410: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240501 14:10:43.218672 6094 raft_consensus.cc:2932] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER be56773105e744919601127a94e98adf (127.29.124.132) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: NON_VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { promote: true } } }
I20240501 14:10:43.220400 5276 tablet_copy_service.cc:342] P 1c3fdc5272484ac09ef3e5e36a08f27c: Request end of tablet copy session 04088ee3bea440a7bd5ff772152b9410-404e2851055541f0be1bb0971a7cc72a received from {username='slave'} at 127.29.124.129:42285
I20240501 14:10:43.220490 5276 tablet_copy_service.cc:433] P 1c3fdc5272484ac09ef3e5e36a08f27c: ending tablet copy session 04088ee3bea440a7bd5ff772152b9410-404e2851055541f0be1bb0971a7cc72a on tablet 404e2851055541f0be1bb0971a7cc72a with peer 04088ee3bea440a7bd5ff772152b9410
I20240501 14:10:43.222142 6125 tablet_copy_client.cc:523] T ac4b234f794149e4b6def7a3b358942d P be56773105e744919601127a94e98adf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:43.223155 5254 raft_consensus.cc:2932] T 63650547ab444f45bb377bcb49f258ad P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER be56773105e744919601127a94e98adf (127.29.124.132) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: NON_VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { promote: true } } }
I20240501 14:10:43.223850 4839 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 63650547ab444f45bb377bcb49f258ad with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:10:43.225925 4852 catalog_manager.cc:5484] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 reported cstate change: config changed from index 6 to 7, NON_VOTER be56773105e744919601127a94e98adf (127.29.124.132) added. New cstate: current_term: 1 leader_uuid: "04088ee3bea440a7bd5ff772152b9410" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: NON_VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:10:43.226536 6125 tablet_bootstrap.cc:492] T ac4b234f794149e4b6def7a3b358942d P be56773105e744919601127a94e98adf: Bootstrap starting.
I20240501 14:10:43.256484 5989 consensus_queue.cc:579] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf [LEADER]: Leader has been unable to successfully communicate with peer 938a24b0e6a040b7966225af26558ff8 for more than 300 seconds (300.108s)
I20240501 14:10:43.257731 6044 consensus_queue.cc:579] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Leader has been unable to successfully communicate with peer 938a24b0e6a040b7966225af26558ff8 for more than 300 seconds (300.181s)
I20240501 14:10:43.263100 5125 consensus_queue.cc:237] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } }
I20240501 14:10:43.265149 5384 consensus_queue.cc:237] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } }
I20240501 14:10:43.268743 6125 tablet_bootstrap.cc:492] T ac4b234f794149e4b6def7a3b358942d P be56773105e744919601127a94e98adf: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:43.269194 6125 tablet_bootstrap.cc:492] T ac4b234f794149e4b6def7a3b358942d P be56773105e744919601127a94e98adf: Bootstrap complete.
I20240501 14:10:43.269551 6125 ts_tablet_manager.cc:1367] T ac4b234f794149e4b6def7a3b358942d P be56773105e744919601127a94e98adf: Time spent bootstrapping tablet: real 0.043s user 0.018s sys 0.006s
I20240501 14:10:43.270321 6125 raft_consensus.cc:348] T ac4b234f794149e4b6def7a3b358942d P be56773105e744919601127a94e98adf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: NON_VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { promote: true } }
I20240501 14:10:43.270563 6125 raft_consensus.cc:729] T ac4b234f794149e4b6def7a3b358942d P be56773105e744919601127a94e98adf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: be56773105e744919601127a94e98adf, State: Initialized, Role: LEARNER
I20240501 14:10:43.270746 6125 consensus_queue.cc:260] T ac4b234f794149e4b6def7a3b358942d P be56773105e744919601127a94e98adf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: NON_VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { promote: true } }
I20240501 14:10:43.271481 6125 ts_tablet_manager.cc:1398] T ac4b234f794149e4b6def7a3b358942d P be56773105e744919601127a94e98adf: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240501 14:10:43.272174 5146 tablet_copy_service.cc:342] P e9c60449cf3c46f8aca4ab6c535302a4: Request end of tablet copy session be56773105e744919601127a94e98adf-ac4b234f794149e4b6def7a3b358942d received from {username='slave'} at 127.29.124.132:43515
I20240501 14:10:43.272208 5990 consensus_queue.cc:579] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Leader has been unable to successfully communicate with peer 938a24b0e6a040b7966225af26558ff8 for more than 300 seconds (300.308s)
I20240501 14:10:43.272285 5146 tablet_copy_service.cc:433] P e9c60449cf3c46f8aca4ab6c535302a4: ending tablet copy session be56773105e744919601127a94e98adf-ac4b234f794149e4b6def7a3b358942d on tablet ac4b234f794149e4b6def7a3b358942d with peer be56773105e744919601127a94e98adf
I20240501 14:10:43.277825 5254 consensus_queue.cc:237] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } }
W20240501 14:10:43.281088 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:43.283279 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:43.286281 4993 raft_consensus.cc:1261] T b284aa82594a44f5885f32fa05844913 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:10:43.286772 5989 consensus_queue.cc:1035] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:10:43.287711 5125 raft_consensus.cc:1261] T b284aa82594a44f5885f32fa05844913 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:10:43.288276 5989 consensus_queue.cc:1035] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:10:43.301638 5989 raft_consensus.cc:2932] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } }
I20240501 14:10:43.306495 4839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet b284aa82594a44f5885f32fa05844913 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:10:43.306206 5126 raft_consensus.cc:2932] T b284aa82594a44f5885f32fa05844913 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } }
I20240501 14:10:43.311602 4852 catalog_manager.cc:5484] T b284aa82594a44f5885f32fa05844913 P e9c60449cf3c46f8aca4ab6c535302a4 reported cstate change: config changed from index 7 to 8, VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) evicted. New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } }
I20240501 14:10:43.315950 5041 maintenance_manager.cc:419] P 04088ee3bea440a7bd5ff772152b9410: Scheduling UndoDeltaBlockGCOp(404e2851055541f0be1bb0971a7cc72a): 4834 bytes on disk
I20240501 14:10:43.314538 4993 raft_consensus.cc:2932] T b284aa82594a44f5885f32fa05844913 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } }
I20240501 14:10:43.316563 4932 maintenance_manager.cc:643] P 04088ee3bea440a7bd5ff772152b9410: UndoDeltaBlockGCOp(404e2851055541f0be1bb0971a7cc72a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4}
I20240501 14:10:43.318691 5384 consensus_queue.cc:237] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } }
I20240501 14:10:43.323042 4993 raft_consensus.cc:1261] T b284aa82594a44f5885f32fa05844913 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:10:43.324126 5126 raft_consensus.cc:1261] T b284aa82594a44f5885f32fa05844913 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:10:43.324792 5727 consensus_queue.cc:1035] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:10:43.325947 5727 consensus_queue.cc:1035] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:10:43.330111 6000 raft_consensus.cc:2932] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:10:43.330662 5126 raft_consensus.cc:2932] T b284aa82594a44f5885f32fa05844913 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:10:43.334937 4839 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet b284aa82594a44f5885f32fa05844913 with cas_config_opid_index 8: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:10:43.335785 4852 catalog_manager.cc:5484] T b284aa82594a44f5885f32fa05844913 P e9c60449cf3c46f8aca4ab6c535302a4 reported cstate change: config changed from index 8 to 9, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New cstate: current_term: 1 leader_uuid: "be56773105e744919601127a94e98adf" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:10:43.340270 6000 consensus_queue.cc:579] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf [LEADER]: Leader has been unable to successfully communicate with peer 938a24b0e6a040b7966225af26558ff8 for more than 300 seconds (300.191s)
W20240501 14:10:43.342605 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240501 14:10:43.343032 4993 raft_consensus.cc:2932] T b284aa82594a44f5885f32fa05844913 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:10:43.353516 5384 consensus_queue.cc:237] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } }
W20240501 14:10:43.406284 5322 connection.cc:535] client connection to 127.29.124.136:43377 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:10:43.407917 5192 connection.cc:535] client connection to 127.29.124.136:43377 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:10:43.409719 5062 connection.cc:535] client connection to 127.29.124.136:43377 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:10:43.412564 6039 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 14:10:43.409143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.124.136:43377 (local address 127.29.124.131:51385)
0501 14:10:43.409524 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0501 14:10:43.409541 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0501 14:10:43.409661 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 14:10:43.412045 (+ 2384us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":261}
W20240501 14:10:43.412580 4815 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 14:10:43.407544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.124.136:43377 (local address 127.29.124.132:46881)
0501 14:10:43.407702 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0501 14:10:43.407722 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0501 14:10:43.407856 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 14:10:43.412046 (+ 4190us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":37}
W20240501 14:10:43.412748 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:43.412837 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:43.412894 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:10:43.412940 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:43.413136 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:10:43.413358 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:43.413470 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:43.414584 4373 negotiation.cc:337] Failed RPC negotiation. Trace:
0501 14:10:43.410734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.29.124.136:43377 (local address 127.29.124.130:37747)
0501 14:10:43.411048 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0501 14:10:43.411066 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0501 14:10:43.411184 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0501 14:10:43.414011 (+ 2827us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":201}
W20240501 14:10:43.415096 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:43.415215 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:43.415308 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:43.415397 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:43.423458 5432 maintenance_manager.cc:419] P be56773105e744919601127a94e98adf: Scheduling UndoDeltaBlockGCOp(ac4b234f794149e4b6def7a3b358942d): 4921 bytes on disk
I20240501 14:10:43.424149 5323 maintenance_manager.cc:643] P be56773105e744919601127a94e98adf: UndoDeltaBlockGCOp(ac4b234f794149e4b6def7a3b358942d) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":111,"lbm_reads_lt_1ms":4}
I20240501 14:10:43.433653 6094 raft_consensus.cc:1050] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410: attempting to promote NON_VOTER 1c3fdc5272484ac09ef3e5e36a08f27c to VOTER
I20240501 14:10:43.434620 6094 consensus_queue.cc:237] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { promote: false } }
I20240501 14:10:43.437508 5254 raft_consensus.cc:1261] T 99d958617b7e4ef385b94f4c73f4dd84 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEARNER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:10:43.437944 5384 raft_consensus.cc:1261] T 99d958617b7e4ef385b94f4c73f4dd84 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240501 14:10:43.438413 4930 consensus_peers.cc:485] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:43.439544 5935 consensus_queue.cc:1035] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:10:43.439898 6096 consensus_queue.cc:1035] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:10:43.451995 6094 raft_consensus.cc:2932] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { promote: false } } }
I20240501 14:10:43.453517 5384 raft_consensus.cc:2932] T 99d958617b7e4ef385b94f4c73f4dd84 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { promote: false } } }
I20240501 14:10:43.457415 5254 raft_consensus.cc:2932] T 99d958617b7e4ef385b94f4c73f4dd84 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { promote: false } } }
I20240501 14:10:43.458479 4852 catalog_manager.cc:5484] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 reported cstate change: config changed from index 7 to 8, 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "04088ee3bea440a7bd5ff772152b9410" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:10:43.464608 4993 consensus_queue.cc:237] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { promote: false } }
I20240501 14:10:43.465389 6081 ts_tablet_manager.cc:902] T 61a274b24f53458f88c2cde4f8dbe4f4 P 04088ee3bea440a7bd5ff772152b9410: Initiating tablet copy from peer be56773105e744919601127a94e98adf (127.29.124.132:35459)
I20240501 14:10:43.467239 5254 raft_consensus.cc:1261] T 99d958617b7e4ef385b94f4c73f4dd84 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:10:43.468557 6094 consensus_queue.cc:1035] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:10:43.469506 5384 raft_consensus.cc:1261] T 99d958617b7e4ef385b94f4c73f4dd84 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:10:43.470391 6094 consensus_queue.cc:1035] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:10:43.471014 5935 raft_consensus.cc:2932] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { promote: false } } }
I20240501 14:10:43.471630 5255 raft_consensus.cc:2932] T 99d958617b7e4ef385b94f4c73f4dd84 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { promote: false } } }
I20240501 14:10:43.475062 4839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 99d958617b7e4ef385b94f4c73f4dd84 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:10:43.477649 4847 catalog_manager.cc:5484] T 99d958617b7e4ef385b94f4c73f4dd84 P 04088ee3bea440a7bd5ff772152b9410 reported cstate change: config changed from index 8 to 9, VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) evicted. New cstate: current_term: 1 leader_uuid: "04088ee3bea440a7bd5ff772152b9410" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:10:43.479399 5386 raft_consensus.cc:2932] T 99d958617b7e4ef385b94f4c73f4dd84 P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { promote: false } } }
I20240501 14:10:43.487196 5727 consensus_queue.cc:579] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf [LEADER]: Leader has been unable to successfully communicate with peer 938a24b0e6a040b7966225af26558ff8 for more than 300 seconds (300.339s)
W20240501 14:10:43.487867 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:43.487979 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:43.489362 5126 raft_consensus.cc:1203] T f4d2b8c005b64bbb8e6621da068d15e5 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.11->[1.12-1.12] Dedup: 1.12->[]
I20240501 14:10:43.493629 6081 tablet_copy_client.cc:308] T 61a274b24f53458f88c2cde4f8dbe4f4 P 04088ee3bea440a7bd5ff772152b9410: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.132:35459
I20240501 14:10:43.494395 5406 tablet_copy_service.cc:140] P be56773105e744919601127a94e98adf: Received BeginTabletCopySession request for tablet 61a274b24f53458f88c2cde4f8dbe4f4 from peer 04088ee3bea440a7bd5ff772152b9410 ({username='slave'} at 127.29.124.129:44079)
I20240501 14:10:43.494561 5406 tablet_copy_service.cc:161] P be56773105e744919601127a94e98adf: Beginning new tablet copy session on tablet 61a274b24f53458f88c2cde4f8dbe4f4 from peer 04088ee3bea440a7bd5ff772152b9410 at {username='slave'} at 127.29.124.129:44079: session id = 04088ee3bea440a7bd5ff772152b9410-61a274b24f53458f88c2cde4f8dbe4f4
I20240501 14:10:43.498292 5406 tablet_copy_source_session.cc:205] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf: Tablet Copy: opened 5 blocks and 1 log segments
W20240501 14:10:43.499579 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:43.499950 6081 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 61a274b24f53458f88c2cde4f8dbe4f4. 1 dirs total, 0 dirs full, 0 dirs failed
W20240501 14:10:43.502326 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240501 14:10:43.506162 6199 ts_tablet_manager.cc:902] T e7d5f02ccba34254a4d20951878b1214 P 04088ee3bea440a7bd5ff772152b9410: Initiating tablet copy from peer be56773105e744919601127a94e98adf (127.29.124.132:35459)
I20240501 14:10:43.507216 5386 consensus_queue.cc:237] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } }
I20240501 14:10:43.510093 6081 tablet_copy_client.cc:783] T 61a274b24f53458f88c2cde4f8dbe4f4 P 04088ee3bea440a7bd5ff772152b9410: tablet copy: Starting download of 5 data blocks...
W20240501 14:10:43.511276 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:43.512274 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:43.518563 6199 tablet_copy_client.cc:308] T e7d5f02ccba34254a4d20951878b1214 P 04088ee3bea440a7bd5ff772152b9410: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.132:35459
I20240501 14:10:43.519104 5406 tablet_copy_service.cc:140] P be56773105e744919601127a94e98adf: Received BeginTabletCopySession request for tablet e7d5f02ccba34254a4d20951878b1214 from peer 04088ee3bea440a7bd5ff772152b9410 ({username='slave'} at 127.29.124.129:44079)
I20240501 14:10:43.519234 6081 tablet_copy_client.cc:647] T 61a274b24f53458f88c2cde4f8dbe4f4 P 04088ee3bea440a7bd5ff772152b9410: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:43.519255 5406 tablet_copy_service.cc:161] P be56773105e744919601127a94e98adf: Beginning new tablet copy session on tablet e7d5f02ccba34254a4d20951878b1214 from peer 04088ee3bea440a7bd5ff772152b9410 at {username='slave'} at 127.29.124.129:44079: session id = 04088ee3bea440a7bd5ff772152b9410-e7d5f02ccba34254a4d20951878b1214
I20240501 14:10:43.521065 6081 tablet_copy_client.cc:523] T 61a274b24f53458f88c2cde4f8dbe4f4 P 04088ee3bea440a7bd5ff772152b9410: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:43.522014 5406 tablet_copy_source_session.cc:205] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 14:10:43.525033 6081 tablet_bootstrap.cc:492] T 61a274b24f53458f88c2cde4f8dbe4f4 P 04088ee3bea440a7bd5ff772152b9410: Bootstrap starting.
W20240501 14:10:43.525611 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:43.525882 6199 ts_tablet_manager.cc:1875] T e7d5f02ccba34254a4d20951878b1214 P 04088ee3bea440a7bd5ff772152b9410: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 14:10:43.533064 6199 ts_tablet_manager.cc:1888] T e7d5f02ccba34254a4d20951878b1214 P 04088ee3bea440a7bd5ff772152b9410: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.9
I20240501 14:10:43.533399 6199 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e7d5f02ccba34254a4d20951878b1214. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:43.536401 6199 tablet_copy_client.cc:783] T e7d5f02ccba34254a4d20951878b1214 P 04088ee3bea440a7bd5ff772152b9410: tablet copy: Starting download of 5 data blocks...
I20240501 14:10:43.537861 4993 raft_consensus.cc:1203] T 88ed5438fc934adc9fb98c6cb4476598 P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
W20240501 14:10:43.545101 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:43.548902 5321 connection.cc:535] client connection to 127.29.124.134:40665 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:10:43.548952 5061 connection.cc:535] client connection to 127.29.124.134:40665 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:10:43.548964 5191 connection.cc:535] client connection to 127.29.124.134:40665 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:10:43.549396 4930 connection.cc:535] client connection to 127.29.124.134:40665 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240501 14:10:43.550307 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:43.550407 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:43.550433 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:43.550521 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:43.550602 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:43.550650 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:10:43.550681 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:43.550722 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:43.550783 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:43.551048 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:43.551154 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:43.551246 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:43.554363 6199 tablet_copy_client.cc:647] T e7d5f02ccba34254a4d20951878b1214 P 04088ee3bea440a7bd5ff772152b9410: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:43.555583 6081 tablet_bootstrap.cc:492] T 61a274b24f53458f88c2cde4f8dbe4f4 P 04088ee3bea440a7bd5ff772152b9410: Bootstrap replayed 1/1 log segments. Stats: ops{read=13 overwritten=0 applied=13 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:43.555974 6081 tablet_bootstrap.cc:492] T 61a274b24f53458f88c2cde4f8dbe4f4 P 04088ee3bea440a7bd5ff772152b9410: Bootstrap complete.
I20240501 14:10:43.556190 6199 tablet_copy_client.cc:523] T e7d5f02ccba34254a4d20951878b1214 P 04088ee3bea440a7bd5ff772152b9410: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:43.556286 6081 ts_tablet_manager.cc:1367] T 61a274b24f53458f88c2cde4f8dbe4f4 P 04088ee3bea440a7bd5ff772152b9410: Time spent bootstrapping tablet: real 0.031s user 0.020s sys 0.004s
I20240501 14:10:43.559867 6199 tablet_bootstrap.cc:492] T e7d5f02ccba34254a4d20951878b1214 P 04088ee3bea440a7bd5ff772152b9410: Bootstrap starting.
I20240501 14:10:43.562885 6081 raft_consensus.cc:348] T 61a274b24f53458f88c2cde4f8dbe4f4 P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: NON_VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { promote: true } }
I20240501 14:10:43.563143 6081 raft_consensus.cc:729] T 61a274b24f53458f88c2cde4f8dbe4f4 P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 04088ee3bea440a7bd5ff772152b9410, State: Initialized, Role: LEARNER
I20240501 14:10:43.563335 6081 consensus_queue.cc:260] T 61a274b24f53458f88c2cde4f8dbe4f4 P 04088ee3bea440a7bd5ff772152b9410 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 13, Last appended: 1.13, Last appended by leader: 13, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: NON_VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { promote: true } }
I20240501 14:10:43.563858 6081 ts_tablet_manager.cc:1398] T 61a274b24f53458f88c2cde4f8dbe4f4 P 04088ee3bea440a7bd5ff772152b9410: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240501 14:10:43.564530 5406 tablet_copy_service.cc:342] P be56773105e744919601127a94e98adf: Request end of tablet copy session 04088ee3bea440a7bd5ff772152b9410-61a274b24f53458f88c2cde4f8dbe4f4 received from {username='slave'} at 127.29.124.129:44079
I20240501 14:10:43.564630 5406 tablet_copy_service.cc:433] P be56773105e744919601127a94e98adf: ending tablet copy session 04088ee3bea440a7bd5ff772152b9410-61a274b24f53458f88c2cde4f8dbe4f4 on tablet 61a274b24f53458f88c2cde4f8dbe4f4 with peer 04088ee3bea440a7bd5ff772152b9410
I20240501 14:10:43.568830 5041 maintenance_manager.cc:419] P 04088ee3bea440a7bd5ff772152b9410: Scheduling UndoDeltaBlockGCOp(61a274b24f53458f88c2cde4f8dbe4f4): 4921 bytes on disk
I20240501 14:10:43.569459 4932 maintenance_manager.cc:643] P 04088ee3bea440a7bd5ff772152b9410: UndoDeltaBlockGCOp(61a274b24f53458f88c2cde4f8dbe4f4) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":127,"lbm_reads_lt_1ms":4}
I20240501 14:10:43.579445 6199 tablet_bootstrap.cc:492] T e7d5f02ccba34254a4d20951878b1214 P 04088ee3bea440a7bd5ff772152b9410: Bootstrap replayed 1/1 log segments. Stats: ops{read=15 overwritten=0 applied=15 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:43.579790 6199 tablet_bootstrap.cc:492] T e7d5f02ccba34254a4d20951878b1214 P 04088ee3bea440a7bd5ff772152b9410: Bootstrap complete.
I20240501 14:10:43.579993 6199 ts_tablet_manager.cc:1367] T e7d5f02ccba34254a4d20951878b1214 P 04088ee3bea440a7bd5ff772152b9410: Time spent bootstrapping tablet: real 0.020s user 0.019s sys 0.001s
I20240501 14:10:43.580572 6199 raft_consensus.cc:348] T e7d5f02ccba34254a4d20951878b1214 P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: NON_VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { promote: true } }
I20240501 14:10:43.580782 6199 raft_consensus.cc:729] T e7d5f02ccba34254a4d20951878b1214 P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 04088ee3bea440a7bd5ff772152b9410, State: Initialized, Role: LEARNER
I20240501 14:10:43.580940 6199 consensus_queue.cc:260] T e7d5f02ccba34254a4d20951878b1214 P 04088ee3bea440a7bd5ff772152b9410 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 15, Last appended: 1.15, Last appended by leader: 15, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: NON_VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { promote: true } }
I20240501 14:10:43.581475 6199 ts_tablet_manager.cc:1398] T e7d5f02ccba34254a4d20951878b1214 P 04088ee3bea440a7bd5ff772152b9410: Time spent starting tablet: real 0.001s user 0.000s sys 0.004s
I20240501 14:10:43.582366 5406 tablet_copy_service.cc:342] P be56773105e744919601127a94e98adf: Request end of tablet copy session 04088ee3bea440a7bd5ff772152b9410-e7d5f02ccba34254a4d20951878b1214 received from {username='slave'} at 127.29.124.129:44079
I20240501 14:10:43.582511 5406 tablet_copy_service.cc:433] P be56773105e744919601127a94e98adf: ending tablet copy session 04088ee3bea440a7bd5ff772152b9410-e7d5f02ccba34254a4d20951878b1214 on tablet e7d5f02ccba34254a4d20951878b1214 with peer 04088ee3bea440a7bd5ff772152b9410
I20240501 14:10:43.591315 6125 ts_tablet_manager.cc:902] T 63650547ab444f45bb377bcb49f258ad P be56773105e744919601127a94e98adf: Initiating tablet copy from peer 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129:44033)
I20240501 14:10:43.592607 6125 tablet_copy_client.cc:308] T 63650547ab444f45bb377bcb49f258ad P be56773105e744919601127a94e98adf: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.129:44033
I20240501 14:10:43.593334 5015 tablet_copy_service.cc:140] P 04088ee3bea440a7bd5ff772152b9410: Received BeginTabletCopySession request for tablet 63650547ab444f45bb377bcb49f258ad from peer be56773105e744919601127a94e98adf ({username='slave'} at 127.29.124.132:46147)
I20240501 14:10:43.593474 5015 tablet_copy_service.cc:161] P 04088ee3bea440a7bd5ff772152b9410: Beginning new tablet copy session on tablet 63650547ab444f45bb377bcb49f258ad from peer be56773105e744919601127a94e98adf at {username='slave'} at 127.29.124.132:46147: session id = be56773105e744919601127a94e98adf-63650547ab444f45bb377bcb49f258ad
I20240501 14:10:43.596295 5015 tablet_copy_source_session.cc:205] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 14:10:43.597738 6125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63650547ab444f45bb377bcb49f258ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:43.602566 5386 raft_consensus.cc:1203] T ac4b234f794149e4b6def7a3b358942d P be56773105e744919601127a94e98adf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
W20240501 14:10:43.605131 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:43.605674 6125 tablet_copy_client.cc:783] T 63650547ab444f45bb377bcb49f258ad P be56773105e744919601127a94e98adf: tablet copy: Starting download of 5 data blocks...
I20240501 14:10:43.613217 6125 tablet_copy_client.cc:647] T 63650547ab444f45bb377bcb49f258ad P be56773105e744919601127a94e98adf: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:43.620326 6125 tablet_copy_client.cc:523] T 63650547ab444f45bb377bcb49f258ad P be56773105e744919601127a94e98adf: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:43.623883 6125 tablet_bootstrap.cc:492] T 63650547ab444f45bb377bcb49f258ad P be56773105e744919601127a94e98adf: Bootstrap starting.
I20240501 14:10:43.623988 6159 consensus_queue.cc:579] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Leader has been unable to successfully communicate with peer 938a24b0e6a040b7966225af26558ff8 for more than 300 seconds (300.415s)
I20240501 14:10:43.628607 5254 consensus_queue.cc:237] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: NON_VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { promote: true } }
I20240501 14:10:43.631569 4993 raft_consensus.cc:1261] T 71660575d94748d8a9329f909a42a0cd P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240501 14:10:43.631731 5190 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377): Couldn't send request to peer e9c60449cf3c46f8aca4ab6c535302a4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 71660575d94748d8a9329f909a42a0cd. This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:43.632133 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:43.632225 6159 consensus_queue.cc:1035] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:10:43.634953 5990 raft_consensus.cc:2932] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: NON_VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { promote: true } } }
I20240501 14:10:43.639190 4838 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 71660575d94748d8a9329f909a42a0cd with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:10:43.639859 4993 raft_consensus.cc:2932] T 71660575d94748d8a9329f909a42a0cd P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: NON_VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { promote: true } } }
I20240501 14:10:43.641054 4853 catalog_manager.cc:5484] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c reported cstate change: config changed from index 9 to 10, NON_VOTER e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130) added. New cstate: current_term: 1 leader_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: NON_VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:10:43.643393 6125 tablet_bootstrap.cc:492] T 63650547ab444f45bb377bcb49f258ad P be56773105e744919601127a94e98adf: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:43.643764 6125 tablet_bootstrap.cc:492] T 63650547ab444f45bb377bcb49f258ad P be56773105e744919601127a94e98adf: Bootstrap complete.
I20240501 14:10:43.644007 6125 ts_tablet_manager.cc:1367] T 63650547ab444f45bb377bcb49f258ad P be56773105e744919601127a94e98adf: Time spent bootstrapping tablet: real 0.020s user 0.019s sys 0.000s
I20240501 14:10:43.644701 6125 raft_consensus.cc:348] T 63650547ab444f45bb377bcb49f258ad P be56773105e744919601127a94e98adf [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: NON_VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { promote: true } }
I20240501 14:10:43.644930 6125 raft_consensus.cc:729] T 63650547ab444f45bb377bcb49f258ad P be56773105e744919601127a94e98adf [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: be56773105e744919601127a94e98adf, State: Initialized, Role: LEARNER
I20240501 14:10:43.645179 6125 consensus_queue.cc:260] T 63650547ab444f45bb377bcb49f258ad P be56773105e744919601127a94e98adf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: NON_VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { promote: true } }
I20240501 14:10:43.648547 6125 ts_tablet_manager.cc:1398] T 63650547ab444f45bb377bcb49f258ad P be56773105e744919601127a94e98adf: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20240501 14:10:43.650158 5015 tablet_copy_service.cc:342] P 04088ee3bea440a7bd5ff772152b9410: Request end of tablet copy session be56773105e744919601127a94e98adf-63650547ab444f45bb377bcb49f258ad received from {username='slave'} at 127.29.124.132:46147
I20240501 14:10:43.650264 5015 tablet_copy_service.cc:433] P 04088ee3bea440a7bd5ff772152b9410: ending tablet copy session be56773105e744919601127a94e98adf-63650547ab444f45bb377bcb49f258ad on tablet 63650547ab444f45bb377bcb49f258ad with peer be56773105e744919601127a94e98adf
W20240501 14:10:43.651818 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:43.653314 5935 consensus_queue.cc:579] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Leader has been unable to successfully communicate with peer 938a24b0e6a040b7966225af26558ff8 for more than 300 seconds (300.569s)
I20240501 14:10:43.659700 4993 raft_consensus.cc:1203] T 404e2851055541f0be1bb0971a7cc72a P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.8->[1.9-1.9] Dedup: 1.9->[]
I20240501 14:10:43.661478 4993 consensus_queue.cc:237] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } }
W20240501 14:10:43.673556 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
I20240501 14:10:43.676602 5432 maintenance_manager.cc:419] P be56773105e744919601127a94e98adf: Scheduling UndoDeltaBlockGCOp(63650547ab444f45bb377bcb49f258ad): 4921 bytes on disk
I20240501 14:10:43.677378 5323 maintenance_manager.cc:643] P be56773105e744919601127a94e98adf: UndoDeltaBlockGCOp(63650547ab444f45bb377bcb49f258ad) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4}
W20240501 14:10:43.683211 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:43.685667 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:43.689823 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:43.691422 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:43.722295 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:43.733868 5893 consensus_queue.cc:579] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Leader has been unable to successfully communicate with peer 938a24b0e6a040b7966225af26558ff8 for more than 300 seconds (300.513s)
I20240501 14:10:43.737417 5126 consensus_queue.cc:237] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: NON_VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { promote: true } }
W20240501 14:10:43.739768 5059 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209): Couldn't send request to peer 1c3fdc5272484ac09ef3e5e36a08f27c. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:43.739969 4993 raft_consensus.cc:1261] T c8a961e711114f369ba06840c062b982 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
W20240501 14:10:43.740178 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:43.740556 5893 consensus_queue.cc:1035] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:10:43.742877 6002 raft_consensus.cc:2932] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: NON_VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { promote: true } } }
I20240501 14:10:43.743420 4993 raft_consensus.cc:2932] T c8a961e711114f369ba06840c062b982 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, NON_VOTER 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) added. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: NON_VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { promote: true } } }
I20240501 14:10:43.745726 4839 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet c8a961e711114f369ba06840c062b982 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:10:43.747650 4847 catalog_manager.cc:5484] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 reported cstate change: config changed from index 9 to 10, NON_VOTER 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) added. New cstate: current_term: 1 leader_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: NON_VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:10:43.821431 5041 maintenance_manager.cc:419] P 04088ee3bea440a7bd5ff772152b9410: Scheduling UndoDeltaBlockGCOp(e7d5f02ccba34254a4d20951878b1214): 4834 bytes on disk
I20240501 14:10:43.821990 4932 maintenance_manager.cc:643] P 04088ee3bea440a7bd5ff772152b9410: UndoDeltaBlockGCOp(e7d5f02ccba34254a4d20951878b1214) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":68,"lbm_reads_lt_1ms":4}
W20240501 14:10:43.829842 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:10:43.883460 4840 catalog_manager.cc:4573] TS 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 99d958617b7e4ef385b94f4c73f4dd84: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:43.911370 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
I20240501 14:10:43.951177 5893 raft_consensus.cc:1050] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4: attempting to promote NON_VOTER 04088ee3bea440a7bd5ff772152b9410 to VOTER
I20240501 14:10:43.952062 5893 consensus_queue.cc:237] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { promote: false } }
I20240501 14:10:43.954078 4993 raft_consensus.cc:1261] T 88ed5438fc934adc9fb98c6cb4476598 P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEARNER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:10:43.954171 5254 raft_consensus.cc:1261] T 88ed5438fc934adc9fb98c6cb4476598 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:10:43.954592 6002 consensus_queue.cc:1035] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:10:43.954816 5893 consensus_queue.cc:1035] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240501 14:10:43.956228 5061 consensus_peers.cc:485] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:43.966362 5893 raft_consensus.cc:2932] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { promote: false } } }
I20240501 14:10:43.966935 5254 raft_consensus.cc:2932] T 88ed5438fc934adc9fb98c6cb4476598 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { promote: false } } }
I20240501 14:10:43.969849 4993 raft_consensus.cc:2932] T 88ed5438fc934adc9fb98c6cb4476598 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { promote: false } } }
I20240501 14:10:43.971165 4853 catalog_manager.cc:5484] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 reported cstate change: config changed from index 7 to 8, 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:10:43.976383 5126 consensus_queue.cc:237] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { promote: false } }
I20240501 14:10:43.978016 4993 raft_consensus.cc:1261] T 88ed5438fc934adc9fb98c6cb4476598 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:10:43.978089 5254 raft_consensus.cc:1261] T 88ed5438fc934adc9fb98c6cb4476598 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:10:43.978518 5893 consensus_queue.cc:1035] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:10:43.978760 6229 consensus_queue.cc:1035] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:10:43.980724 6002 raft_consensus.cc:2932] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { promote: false } } }
I20240501 14:10:43.981241 5254 raft_consensus.cc:2932] T 88ed5438fc934adc9fb98c6cb4476598 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { promote: false } } }
I20240501 14:10:43.981289 4993 raft_consensus.cc:2932] T 88ed5438fc934adc9fb98c6cb4476598 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { promote: false } } }
I20240501 14:10:43.984927 4839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 88ed5438fc934adc9fb98c6cb4476598 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:10:43.986394 4847 catalog_manager.cc:5484] T 88ed5438fc934adc9fb98c6cb4476598 P e9c60449cf3c46f8aca4ab6c535302a4 reported cstate change: config changed from index 8 to 9, VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) evicted. New cstate: current_term: 1 leader_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:10:44.022899 6002 consensus_queue.cc:579] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Leader has been unable to successfully communicate with peer 4360b59e3a154688a79ca5f9c0a37bc9 for more than 300 seconds (300.171s)
I20240501 14:10:44.026528 4993 raft_consensus.cc:1203] T 61a274b24f53458f88c2cde4f8dbe4f4 P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.12->[1.13-1.13] Dedup: 1.13->[]
I20240501 14:10:44.026523 5386 raft_consensus.cc:1203] T 63650547ab444f45bb377bcb49f258ad P be56773105e744919601127a94e98adf [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240501 14:10:44.028028 5126 consensus_queue.cc:237] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 2.13, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } }
W20240501 14:10:44.030918 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:44.031101 4993 raft_consensus.cc:1261] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [term 2 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 13. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20240501 14:10:44.031582 6002 consensus_queue.cc:1035] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
I20240501 14:10:44.033740 6044 raft_consensus.cc:2932] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 [term 2 LEADER]: Committing config change with OpId 2.14: config changed from index 12 to 14, VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:10:44.034379 4993 raft_consensus.cc:2932] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [term 2 FOLLOWER]: Committing config change with OpId 2.14: config changed from index 12 to 14, VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) evicted. New config: { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } }
I20240501 14:10:44.036623 4839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fee16905387d428aa02cd2070c86c790 with cas_config_opid_index 12: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:10:44.038058 4850 catalog_manager.cc:5484] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 reported cstate change: config changed from index 12 to 14, VOTER 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135) evicted. New cstate: current_term: 2 leader_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" committed_config { opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240501 14:10:44.042927 5126 consensus_queue.cc:237] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } }
I20240501 14:10:44.045571 4993 raft_consensus.cc:1261] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [term 2 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 14. Preceding OpId from leader: term: 2 index: 15. (index mismatch)
I20240501 14:10:44.046099 5893 consensus_queue.cc:1035] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 15, Last known committed idx: 14, Time since last communication: 0.000s
W20240501 14:10:44.047076 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:44.047436 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:44.048144 6002 raft_consensus.cc:2932] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 [term 2 LEADER]: Committing config change with OpId 2.15: config changed from index 14 to 15, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:10:44.048789 4993 raft_consensus.cc:2932] T fee16905387d428aa02cd2070c86c790 P 04088ee3bea440a7bd5ff772152b9410 [term 2 FOLLOWER]: Committing config change with OpId 2.15: config changed from index 14 to 15, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New config: { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } } }
I20240501 14:10:44.052498 4839 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet fee16905387d428aa02cd2070c86c790 with cas_config_opid_index 14: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:10:44.053432 4850 catalog_manager.cc:5484] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 reported cstate change: config changed from index 14 to 15, NON_VOTER 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136) added. New cstate: current_term: 2 leader_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" committed_config { opid_index: 15 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240501 14:10:44.054746 6049 raft_consensus.cc:1050] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c: attempting to promote NON_VOTER 04088ee3bea440a7bd5ff772152b9410 to VOTER
I20240501 14:10:44.055466 6049 consensus_queue.cc:237] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { promote: false } }
I20240501 14:10:44.060312 5126 raft_consensus.cc:1261] T 404e2851055541f0be1bb0971a7cc72a P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:10:44.060356 4993 raft_consensus.cc:1261] T 404e2851055541f0be1bb0971a7cc72a P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEARNER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20240501 14:10:44.060952 6159 consensus_queue.cc:1035] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240501 14:10:44.061329 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:44.061513 6159 consensus_queue.cc:1035] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:10:44.066613 6049 raft_consensus.cc:2932] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { promote: false } } }
I20240501 14:10:44.070025 4993 raft_consensus.cc:2932] T 404e2851055541f0be1bb0971a7cc72a P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { promote: false } } }
I20240501 14:10:44.071447 5126 raft_consensus.cc:2932] T 404e2851055541f0be1bb0971a7cc72a P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 9 to 10, 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { promote: false } } }
I20240501 14:10:44.071669 4852 catalog_manager.cc:5484] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c reported cstate change: config changed from index 9 to 10, 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:10:44.075564 4994 raft_consensus.cc:1203] T e7d5f02ccba34254a4d20951878b1214 P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.14->[1.15-1.15] Dedup: 1.15->[]
I20240501 14:10:44.097541 5988 raft_consensus.cc:1050] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf: attempting to promote NON_VOTER e9c60449cf3c46f8aca4ab6c535302a4 to VOTER
I20240501 14:10:44.098367 5988 consensus_queue.cc:237] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { promote: false } }
I20240501 14:10:44.100919 5126 raft_consensus.cc:1261] T f4d2b8c005b64bbb8e6621da068d15e5 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 LEARNER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 1 index: 13. (index mismatch)
I20240501 14:10:44.101792 6255 consensus_queue.cc:1035] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20240501 14:10:44.103094 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:44.103255 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:44.124464 6265 ts_tablet_manager.cc:902] T 71660575d94748d8a9329f909a42a0cd P e9c60449cf3c46f8aca4ab6c535302a4: Initiating tablet copy from peer 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209)
I20240501 14:10:44.127352 6265 tablet_copy_client.cc:308] T 71660575d94748d8a9329f909a42a0cd P e9c60449cf3c46f8aca4ab6c535302a4: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.131:33209
I20240501 14:10:44.127892 5276 tablet_copy_service.cc:140] P 1c3fdc5272484ac09ef3e5e36a08f27c: Received BeginTabletCopySession request for tablet 71660575d94748d8a9329f909a42a0cd from peer e9c60449cf3c46f8aca4ab6c535302a4 ({username='slave'} at 127.29.124.130:54039)
I20240501 14:10:44.128016 5276 tablet_copy_service.cc:161] P 1c3fdc5272484ac09ef3e5e36a08f27c: Beginning new tablet copy session on tablet 71660575d94748d8a9329f909a42a0cd from peer e9c60449cf3c46f8aca4ab6c535302a4 at {username='slave'} at 127.29.124.130:54039: session id = e9c60449cf3c46f8aca4ab6c535302a4-71660575d94748d8a9329f909a42a0cd
I20240501 14:10:44.130532 5276 tablet_copy_source_session.cc:205] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 14:10:44.131736 6265 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 71660575d94748d8a9329f909a42a0cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:44.136168 6265 tablet_copy_client.cc:783] T 71660575d94748d8a9329f909a42a0cd P e9c60449cf3c46f8aca4ab6c535302a4: tablet copy: Starting download of 5 data blocks...
I20240501 14:10:44.142602 6265 tablet_copy_client.cc:647] T 71660575d94748d8a9329f909a42a0cd P e9c60449cf3c46f8aca4ab6c535302a4: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:44.144120 6265 tablet_copy_client.cc:523] T 71660575d94748d8a9329f909a42a0cd P e9c60449cf3c46f8aca4ab6c535302a4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:44.147408 6265 tablet_bootstrap.cc:492] T 71660575d94748d8a9329f909a42a0cd P e9c60449cf3c46f8aca4ab6c535302a4: Bootstrap starting.
I20240501 14:10:44.163568 6265 tablet_bootstrap.cc:492] T 71660575d94748d8a9329f909a42a0cd P e9c60449cf3c46f8aca4ab6c535302a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:44.163882 6265 tablet_bootstrap.cc:492] T 71660575d94748d8a9329f909a42a0cd P e9c60449cf3c46f8aca4ab6c535302a4: Bootstrap complete.
I20240501 14:10:44.164077 6265 ts_tablet_manager.cc:1367] T 71660575d94748d8a9329f909a42a0cd P e9c60449cf3c46f8aca4ab6c535302a4: Time spent bootstrapping tablet: real 0.017s user 0.016s sys 0.000s
I20240501 14:10:44.164610 6265 raft_consensus.cc:348] T 71660575d94748d8a9329f909a42a0cd P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: NON_VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { promote: true } }
I20240501 14:10:44.164779 6265 raft_consensus.cc:729] T 71660575d94748d8a9329f909a42a0cd P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e9c60449cf3c46f8aca4ab6c535302a4, State: Initialized, Role: LEARNER
I20240501 14:10:44.164920 6265 consensus_queue.cc:260] T 71660575d94748d8a9329f909a42a0cd P e9c60449cf3c46f8aca4ab6c535302a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: NON_VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { promote: true } }
I20240501 14:10:44.165369 6265 ts_tablet_manager.cc:1398] T 71660575d94748d8a9329f909a42a0cd P e9c60449cf3c46f8aca4ab6c535302a4: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240501 14:10:44.166070 5276 tablet_copy_service.cc:342] P 1c3fdc5272484ac09ef3e5e36a08f27c: Request end of tablet copy session e9c60449cf3c46f8aca4ab6c535302a4-71660575d94748d8a9329f909a42a0cd received from {username='slave'} at 127.29.124.130:54039
I20240501 14:10:44.166200 5276 tablet_copy_service.cc:433] P 1c3fdc5272484ac09ef3e5e36a08f27c: ending tablet copy session e9c60449cf3c46f8aca4ab6c535302a4-71660575d94748d8a9329f909a42a0cd on tablet 71660575d94748d8a9329f909a42a0cd with peer e9c60449cf3c46f8aca4ab6c535302a4
W20240501 14:10:44.192827 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
I20240501 14:10:44.214319 5893 raft_consensus.cc:1050] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4: attempting to promote NON_VOTER be56773105e744919601127a94e98adf to VOTER
I20240501 14:10:44.215027 5893 consensus_queue.cc:237] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { promote: false } }
I20240501 14:10:44.217360 5386 raft_consensus.cc:1261] T ac4b234f794149e4b6def7a3b358942d P be56773105e744919601127a94e98adf [term 1 LEARNER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:10:44.217931 6044 consensus_queue.cc:1035] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240501 14:10:44.218206 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:44.218212 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:44.220794 5172 maintenance_manager.cc:419] P e9c60449cf3c46f8aca4ab6c535302a4: Scheduling UndoDeltaBlockGCOp(71660575d94748d8a9329f909a42a0cd): 4581 bytes on disk
I20240501 14:10:44.221319 5063 maintenance_manager.cc:643] P e9c60449cf3c46f8aca4ab6c535302a4: UndoDeltaBlockGCOp(71660575d94748d8a9329f909a42a0cd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
I20240501 14:10:44.225795 6274 ts_tablet_manager.cc:902] T c8a961e711114f369ba06840c062b982 P 1c3fdc5272484ac09ef3e5e36a08f27c: Initiating tablet copy from peer e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377)
I20240501 14:10:44.227005 6274 tablet_copy_client.cc:308] T c8a961e711114f369ba06840c062b982 P 1c3fdc5272484ac09ef3e5e36a08f27c: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.130:34377
I20240501 14:10:44.227560 5146 tablet_copy_service.cc:140] P e9c60449cf3c46f8aca4ab6c535302a4: Received BeginTabletCopySession request for tablet c8a961e711114f369ba06840c062b982 from peer 1c3fdc5272484ac09ef3e5e36a08f27c ({username='slave'} at 127.29.124.131:42095)
I20240501 14:10:44.227681 5146 tablet_copy_service.cc:161] P e9c60449cf3c46f8aca4ab6c535302a4: Beginning new tablet copy session on tablet c8a961e711114f369ba06840c062b982 from peer 1c3fdc5272484ac09ef3e5e36a08f27c at {username='slave'} at 127.29.124.131:42095: session id = 1c3fdc5272484ac09ef3e5e36a08f27c-c8a961e711114f369ba06840c062b982
I20240501 14:10:44.229794 5146 tablet_copy_source_session.cc:205] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 14:10:44.230763 6274 ts_tablet_manager.cc:1875] T c8a961e711114f369ba06840c062b982 P 1c3fdc5272484ac09ef3e5e36a08f27c: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 14:10:44.235286 6274 ts_tablet_manager.cc:1888] T c8a961e711114f369ba06840c062b982 P 1c3fdc5272484ac09ef3e5e36a08f27c: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.7
I20240501 14:10:44.235543 6274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c8a961e711114f369ba06840c062b982. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:44.237686 6274 tablet_copy_client.cc:783] T c8a961e711114f369ba06840c062b982 P 1c3fdc5272484ac09ef3e5e36a08f27c: tablet copy: Starting download of 5 data blocks...
I20240501 14:10:44.242667 6274 tablet_copy_client.cc:647] T c8a961e711114f369ba06840c062b982 P 1c3fdc5272484ac09ef3e5e36a08f27c: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:44.244124 6274 tablet_copy_client.cc:523] T c8a961e711114f369ba06840c062b982 P 1c3fdc5272484ac09ef3e5e36a08f27c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:44.246599 6274 tablet_bootstrap.cc:492] T c8a961e711114f369ba06840c062b982 P 1c3fdc5272484ac09ef3e5e36a08f27c: Bootstrap starting.
I20240501 14:10:44.260120 6274 tablet_bootstrap.cc:492] T c8a961e711114f369ba06840c062b982 P 1c3fdc5272484ac09ef3e5e36a08f27c: Bootstrap replayed 1/1 log segments. Stats: ops{read=10 overwritten=0 applied=10 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:44.260412 6274 tablet_bootstrap.cc:492] T c8a961e711114f369ba06840c062b982 P 1c3fdc5272484ac09ef3e5e36a08f27c: Bootstrap complete.
I20240501 14:10:44.260612 6274 ts_tablet_manager.cc:1367] T c8a961e711114f369ba06840c062b982 P 1c3fdc5272484ac09ef3e5e36a08f27c: Time spent bootstrapping tablet: real 0.014s user 0.010s sys 0.002s
I20240501 14:10:44.261132 6274 raft_consensus.cc:348] T c8a961e711114f369ba06840c062b982 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: NON_VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { promote: true } }
I20240501 14:10:44.261364 6274 raft_consensus.cc:729] T c8a961e711114f369ba06840c062b982 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1c3fdc5272484ac09ef3e5e36a08f27c, State: Initialized, Role: LEARNER
I20240501 14:10:44.261561 6274 consensus_queue.cc:260] T c8a961e711114f369ba06840c062b982 P 1c3fdc5272484ac09ef3e5e36a08f27c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 10, Last appended: 1.10, Last appended by leader: 10, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: NON_VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { promote: true } }
I20240501 14:10:44.262054 6274 ts_tablet_manager.cc:1398] T c8a961e711114f369ba06840c062b982 P 1c3fdc5272484ac09ef3e5e36a08f27c: Time spent starting tablet: real 0.001s user 0.004s sys 0.000s
I20240501 14:10:44.262924 5146 tablet_copy_service.cc:342] P e9c60449cf3c46f8aca4ab6c535302a4: Request end of tablet copy session 1c3fdc5272484ac09ef3e5e36a08f27c-c8a961e711114f369ba06840c062b982 received from {username='slave'} at 127.29.124.131:42095
I20240501 14:10:44.263038 5146 tablet_copy_service.cc:433] P e9c60449cf3c46f8aca4ab6c535302a4: ending tablet copy session 1c3fdc5272484ac09ef3e5e36a08f27c-c8a961e711114f369ba06840c062b982 on tablet c8a961e711114f369ba06840c062b982 with peer 1c3fdc5272484ac09ef3e5e36a08f27c
W20240501 14:10:44.288795 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:10:44.289364 5191 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240501 14:10:44.349637 6255 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:44.349818 6255 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:10:44.350713 6255 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:10:44.351105 5254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" is_pre_election: true
I20240501 14:10:44.351361 5254 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 2.
W20240501 14:10:44.352082 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:44.352221 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:10:44.352293 5321 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:10:44.352509 6255 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240501 14:10:44.416419 6255 raft_consensus.cc:482] T a726769d24284fedadd8e9e63cd27d05 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 19a6d2d8a08142d5bffcde3a37234692)
I20240501 14:10:44.416626 6255 raft_consensus.cc:504] T a726769d24284fedadd8e9e63cd27d05 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } }
I20240501 14:10:44.417287 6255 leader_election.cc:288] T a726769d24284fedadd8e9e63cd27d05 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377), 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665)
I20240501 14:10:44.417716 5126 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a726769d24284fedadd8e9e63cd27d05" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" is_pre_election: true
W20240501 14:10:44.418566 5321 leader_election.cc:334] T a726769d24284fedadd8e9e63cd27d05 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
I20240501 14:10:44.418680 5321 leader_election.cc:302] T a726769d24284fedadd8e9e63cd27d05 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: be56773105e744919601127a94e98adf; no voters: 19a6d2d8a08142d5bffcde3a37234692, e9c60449cf3c46f8aca4ab6c535302a4
I20240501 14:10:44.418915 6255 raft_consensus.cc:2726] T a726769d24284fedadd8e9e63cd27d05 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:10:44.444232 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
I20240501 14:10:44.473685 6255 raft_consensus.cc:1050] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf: attempting to promote NON_VOTER 04088ee3bea440a7bd5ff772152b9410 to VOTER
I20240501 14:10:44.474431 6255 consensus_queue.cc:237] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 13, Committed index: 13, Last appended: 1.13, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { promote: false } }
I20240501 14:10:44.476665 4994 raft_consensus.cc:1261] T 61a274b24f53458f88c2cde4f8dbe4f4 P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEARNER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 13. Preceding OpId from leader: term: 1 index: 14. (index mismatch)
W20240501 14:10:44.477175 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:44.477311 6255 consensus_queue.cc:1035] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 14, Last known committed idx: 13, Time since last communication: 0.000s
W20240501 14:10:44.477525 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:44.484230 6094 raft_consensus.cc:1050] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410: attempting to promote NON_VOTER be56773105e744919601127a94e98adf to VOTER
I20240501 14:10:44.484841 6094 consensus_queue.cc:237] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { promote: false } }
I20240501 14:10:44.493590 5302 maintenance_manager.cc:419] P 1c3fdc5272484ac09ef3e5e36a08f27c: Scheduling UndoDeltaBlockGCOp(c8a961e711114f369ba06840c062b982): 4834 bytes on disk
I20240501 14:10:44.494285 5193 maintenance_manager.cc:643] P 1c3fdc5272484ac09ef3e5e36a08f27c: UndoDeltaBlockGCOp(c8a961e711114f369ba06840c062b982) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4}
I20240501 14:10:44.497913 5386 raft_consensus.cc:1261] T 63650547ab444f45bb377bcb49f258ad P be56773105e744919601127a94e98adf [term 1 LEARNER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240501 14:10:44.498450 5254 raft_consensus.cc:1261] T 63650547ab444f45bb377bcb49f258ad P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
W20240501 14:10:44.498888 4930 consensus_peers.cc:485] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:44.499171 5935 consensus_queue.cc:1035] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:10:44.499606 6096 consensus_queue.cc:1035] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240501 14:10:44.502987 5126 raft_consensus.cc:1203] T 71660575d94748d8a9329f909a42a0cd P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
I20240501 14:10:44.502992 6096 raft_consensus.cc:2932] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, be56773105e744919601127a94e98adf (127.29.124.132) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { promote: false } } }
I20240501 14:10:44.508260 5386 raft_consensus.cc:2932] T 63650547ab444f45bb377bcb49f258ad P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, be56773105e744919601127a94e98adf (127.29.124.132) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { promote: false } } }
I20240501 14:10:44.509589 5254 raft_consensus.cc:2932] T 63650547ab444f45bb377bcb49f258ad P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, be56773105e744919601127a94e98adf (127.29.124.132) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { promote: false } } }
I20240501 14:10:44.510076 4847 catalog_manager.cc:5484] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 reported cstate change: config changed from index 7 to 8, be56773105e744919601127a94e98adf (127.29.124.132) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "04088ee3bea440a7bd5ff772152b9410" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:10:44.518070 4994 consensus_queue.cc:237] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { promote: false } }
I20240501 14:10:44.520174 5254 raft_consensus.cc:1261] T 63650547ab444f45bb377bcb49f258ad P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:10:44.520758 6094 consensus_queue.cc:1035] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:10:44.524240 6295 raft_consensus.cc:2932] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { promote: false } } }
I20240501 14:10:44.524792 5254 raft_consensus.cc:2932] T 63650547ab444f45bb377bcb49f258ad P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { promote: false } } }
I20240501 14:10:44.528017 4839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 63650547ab444f45bb377bcb49f258ad with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:10:44.529989 4850 catalog_manager.cc:5484] T 63650547ab444f45bb377bcb49f258ad P 1c3fdc5272484ac09ef3e5e36a08f27c reported cstate change: config changed from index 8 to 9, VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) evicted. New cstate: current_term: 1 leader_uuid: "04088ee3bea440a7bd5ff772152b9410" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { promote: false } } }
I20240501 14:10:44.536806 5384 raft_consensus.cc:1261] T 63650547ab444f45bb377bcb49f258ad P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Refusing update from remote peer 04088ee3bea440a7bd5ff772152b9410: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240501 14:10:44.537884 6094 consensus_queue.cc:1035] T 63650547ab444f45bb377bcb49f258ad P 04088ee3bea440a7bd5ff772152b9410 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240501 14:10:44.538172 6255 raft_consensus.cc:1050] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf: attempting to promote NON_VOTER 04088ee3bea440a7bd5ff772152b9410 to VOTER
I20240501 14:10:44.538966 6255 consensus_queue.cc:237] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 15, Committed index: 15, Last appended: 1.15, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 16 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { promote: false } }
I20240501 14:10:44.539644 5384 raft_consensus.cc:2932] T 63650547ab444f45bb377bcb49f258ad P be56773105e744919601127a94e98adf [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { promote: false } } }
I20240501 14:10:44.542691 4994 raft_consensus.cc:1261] T e7d5f02ccba34254a4d20951878b1214 P 04088ee3bea440a7bd5ff772152b9410 [term 1 LEARNER]: Refusing update from remote peer be56773105e744919601127a94e98adf: Log matching property violated. Preceding OpId in replica: term: 1 index: 15. Preceding OpId from leader: term: 1 index: 16. (index mismatch)
I20240501 14:10:44.543282 5727 consensus_queue.cc:1035] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 16, Last known committed idx: 15, Time since last communication: 0.000s
W20240501 14:10:44.543627 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:44.545248 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:44.571976 6044 raft_consensus.cc:482] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 19a6d2d8a08142d5bffcde3a37234692)
I20240501 14:10:44.572172 6044 raft_consensus.cc:504] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } }
I20240501 14:10:44.572767 6044 leader_election.cc:288] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers be56773105e744919601127a94e98adf (127.29.124.132:35459), 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665)
I20240501 14:10:44.573221 5384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a726769d24284fedadd8e9e63cd27d05" candidate_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf" is_pre_election: true
I20240501 14:10:44.573494 5384 raft_consensus.cc:2454] T a726769d24284fedadd8e9e63cd27d05 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e9c60449cf3c46f8aca4ab6c535302a4 in term 2.
I20240501 14:10:44.573899 5060 leader_election.cc:302] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: be56773105e744919601127a94e98adf, e9c60449cf3c46f8aca4ab6c535302a4; no voters:
I20240501 14:10:44.574122 6044 raft_consensus.cc:2781] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240501 14:10:44.574195 6044 raft_consensus.cc:482] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 19a6d2d8a08142d5bffcde3a37234692)
W20240501 14:10:44.574244 5061 leader_election.cc:334] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
I20240501 14:10:44.574263 6044 raft_consensus.cc:3037] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 [term 2 FOLLOWER]: Advancing to term 3
I20240501 14:10:44.576539 6044 raft_consensus.cc:504] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } }
I20240501 14:10:44.577008 6044 leader_election.cc:288] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 [CANDIDATE]: Term 3 election: Requested vote from peers be56773105e744919601127a94e98adf (127.29.124.132:35459), 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665)
I20240501 14:10:44.577390 5384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a726769d24284fedadd8e9e63cd27d05" candidate_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" candidate_term: 3 candidate_status { last_received { term: 2 index: 9 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf"
I20240501 14:10:44.577550 5384 raft_consensus.cc:3037] T a726769d24284fedadd8e9e63cd27d05 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Advancing to term 3
W20240501 14:10:44.578133 5061 leader_election.cc:334] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
I20240501 14:10:44.579411 5384 raft_consensus.cc:2454] T a726769d24284fedadd8e9e63cd27d05 P be56773105e744919601127a94e98adf [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e9c60449cf3c46f8aca4ab6c535302a4 in term 3.
I20240501 14:10:44.579725 5060 leader_election.cc:302] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: be56773105e744919601127a94e98adf, e9c60449cf3c46f8aca4ab6c535302a4; no voters: 19a6d2d8a08142d5bffcde3a37234692
I20240501 14:10:44.579949 6044 raft_consensus.cc:2781] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 [term 3 FOLLOWER]: Leader election won for term 3
I20240501 14:10:44.580067 6044 raft_consensus.cc:686] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 [term 3 LEADER]: Becoming Leader. State: Replica: e9c60449cf3c46f8aca4ab6c535302a4, State: Running, Role: LEADER
I20240501 14:10:44.580307 6044 consensus_queue.cc:237] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 9, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } }
I20240501 14:10:44.584774 4847 catalog_manager.cc:5484] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 reported cstate change: term changed from 2 to 3, leader changed from 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134) to e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130). New cstate: current_term: 3 leader_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240501 14:10:44.632858 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:10:44.661417 5254 raft_consensus.cc:1203] T c8a961e711114f369ba06840c062b982 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.9->[1.10-1.10] Dedup: 1.10->[]
W20240501 14:10:44.675979 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
I20240501 14:10:44.807224 5990 consensus_queue.cc:579] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Leader has been unable to successfully communicate with peer 938a24b0e6a040b7966225af26558ff8 for more than 300 seconds (300.406s)
I20240501 14:10:44.812419 5254 consensus_queue.cc:237] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } }
I20240501 14:10:44.814611 5384 raft_consensus.cc:1261] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
W20240501 14:10:44.814991 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:44.815161 5990 consensus_queue.cc:1035] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240501 14:10:44.817268 6251 raft_consensus.cc:2932] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:10:44.820039 4838 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 420642af520d45ffbc7f2d35f93e75a7 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:10:44.820456 5384 raft_consensus.cc:2932] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } }
I20240501 14:10:44.821646 4850 catalog_manager.cc:5484] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c reported cstate change: config changed from index 8 to 10, VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) evicted. New cstate: current_term: 2 leader_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240501 14:10:44.827992 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
I20240501 14:10:44.828280 5254 consensus_queue.cc:237] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 2.10, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: NON_VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { promote: true } }
I20240501 14:10:44.830379 5384 raft_consensus.cc:1261] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 2 index: 10. Preceding OpId from leader: term: 2 index: 11. (index mismatch)
W20240501 14:10:44.830937 5190 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377): Couldn't send request to peer e9c60449cf3c46f8aca4ab6c535302a4. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:44.831401 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:44.831605 6251 consensus_queue.cc:1035] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:10:44.833748 5990 raft_consensus.cc:2932] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 LEADER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: NON_VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { promote: true } } }
I20240501 14:10:44.834764 5384 raft_consensus.cc:2932] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Committing config change with OpId 2.11: config changed from index 10 to 11, NON_VOTER e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130) added. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: NON_VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { promote: true } } }
I20240501 14:10:44.836578 4843 maintenance_manager.cc:419] P f10c6a9c4bad46deb63190d64169ec11: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 10691 bytes on disk
I20240501 14:10:44.837039 4838 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 420642af520d45ffbc7f2d35f93e75a7 with cas_config_opid_index 10: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240501 14:10:44.838954 4847 catalog_manager.cc:5484] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf reported cstate change: config changed from index 10 to 11, NON_VOTER e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130) added. New cstate: current_term: 2 leader_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: NON_VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { promote: true } } }
I20240501 14:10:44.841996 4842 maintenance_manager.cc:643] P f10c6a9c4bad46deb63190d64169ec11: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.005s user 0.004s sys 0.000s Metrics: {"dirs.queue_time_us":337,"dirs.run_cpu_time_us":97,"dirs.run_wall_time_us":97,"thread_start_us":384,"threads_started":1}
W20240501 14:10:44.874928 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
I20240501 14:10:44.907938 6049 raft_consensus.cc:1050] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c: attempting to promote NON_VOTER e9c60449cf3c46f8aca4ab6c535302a4 to VOTER
I20240501 14:10:44.908826 6049 consensus_queue.cc:237] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { promote: false } }
W20240501 14:10:44.909485 4840 catalog_manager.cc:4573] TS 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 420642af520d45ffbc7f2d35f93e75a7: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:10:44.911123 5126 raft_consensus.cc:1261] T 71660575d94748d8a9329f909a42a0cd P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 LEARNER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:10:44.911183 4994 raft_consensus.cc:1261] T 71660575d94748d8a9329f909a42a0cd P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:10:44.911810 6325 consensus_queue.cc:1035] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:10:44.912087 6049 consensus_queue.cc:1035] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240501 14:10:44.912317 5191 consensus_peers.cc:485] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:44.914515 6325 raft_consensus.cc:2932] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { promote: false } } }
I20240501 14:10:44.915140 4994 raft_consensus.cc:2932] T 71660575d94748d8a9329f909a42a0cd P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { promote: false } } }
I20240501 14:10:44.917724 5126 raft_consensus.cc:2932] T 71660575d94748d8a9329f909a42a0cd P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { promote: false } } }
I20240501 14:10:44.919384 4850 catalog_manager.cc:5484] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c reported cstate change: config changed from index 10 to 11, e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:10:44.923740 5254 consensus_queue.cc:237] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { promote: false } }
I20240501 14:10:44.925226 5126 raft_consensus.cc:1261] T 71660575d94748d8a9329f909a42a0cd P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:10:44.925393 4994 raft_consensus.cc:1261] T 71660575d94748d8a9329f909a42a0cd P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:10:44.925848 6251 consensus_queue.cc:1035] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:10:44.926070 5990 consensus_queue.cc:1035] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:10:44.928040 5990 raft_consensus.cc:2932] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { promote: false } } }
I20240501 14:10:44.929841 4994 raft_consensus.cc:2932] T 71660575d94748d8a9329f909a42a0cd P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { promote: false } } }
I20240501 14:10:44.930500 5126 raft_consensus.cc:2932] T 71660575d94748d8a9329f909a42a0cd P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { promote: false } } }
I20240501 14:10:44.931020 4838 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 71660575d94748d8a9329f909a42a0cd with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:10:44.933327 4852 catalog_manager.cc:5484] T 71660575d94748d8a9329f909a42a0cd P 1c3fdc5272484ac09ef3e5e36a08f27c reported cstate change: config changed from index 11 to 12, VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) evicted. New cstate: current_term: 1 leader_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:10:44.976951 5384 raft_consensus.cc:1261] T a726769d24284fedadd8e9e63cd27d05 P be56773105e744919601127a94e98adf [term 3 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 3 index: 10. (index mismatch)
I20240501 14:10:44.977427 6044 consensus_queue.cc:1035] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
W20240501 14:10:44.981707 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240501 14:10:45.055089 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 53 similar messages]
W20240501 14:10:45.177634 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:10:45.180616 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:45.200860 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:10:45.242388 6353 ts_tablet_manager.cc:902] T 420642af520d45ffbc7f2d35f93e75a7 P e9c60449cf3c46f8aca4ab6c535302a4: Initiating tablet copy from peer 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209)
I20240501 14:10:45.243410 6353 tablet_copy_client.cc:308] T 420642af520d45ffbc7f2d35f93e75a7 P e9c60449cf3c46f8aca4ab6c535302a4: tablet copy: Beginning tablet copy session from remote peer at address 127.29.124.131:33209
I20240501 14:10:45.244103 5276 tablet_copy_service.cc:140] P 1c3fdc5272484ac09ef3e5e36a08f27c: Received BeginTabletCopySession request for tablet 420642af520d45ffbc7f2d35f93e75a7 from peer e9c60449cf3c46f8aca4ab6c535302a4 ({username='slave'} at 127.29.124.130:54039)
I20240501 14:10:45.244282 5276 tablet_copy_service.cc:161] P 1c3fdc5272484ac09ef3e5e36a08f27c: Beginning new tablet copy session on tablet 420642af520d45ffbc7f2d35f93e75a7 from peer e9c60449cf3c46f8aca4ab6c535302a4 at {username='slave'} at 127.29.124.130:54039: session id = e9c60449cf3c46f8aca4ab6c535302a4-420642af520d45ffbc7f2d35f93e75a7
I20240501 14:10:45.246932 5276 tablet_copy_source_session.cc:205] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c: Tablet Copy: opened 5 blocks and 1 log segments
I20240501 14:10:45.248153 6353 ts_tablet_manager.cc:1875] T 420642af520d45ffbc7f2d35f93e75a7 P e9c60449cf3c46f8aca4ab6c535302a4: Deleting tablet data with delete state TABLET_DATA_COPYING
I20240501 14:10:45.252547 6353 ts_tablet_manager.cc:1888] T 420642af520d45ffbc7f2d35f93e75a7 P e9c60449cf3c46f8aca4ab6c535302a4: tablet deleted with delete type TABLET_DATA_COPYING: last-logged OpId 1.5
I20240501 14:10:45.252759 6353 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 420642af520d45ffbc7f2d35f93e75a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240501 14:10:45.254709 6353 tablet_copy_client.cc:783] T 420642af520d45ffbc7f2d35f93e75a7 P e9c60449cf3c46f8aca4ab6c535302a4: tablet copy: Starting download of 5 data blocks...
I20240501 14:10:45.258152 6229 raft_consensus.cc:1050] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4: attempting to promote NON_VOTER 1c3fdc5272484ac09ef3e5e36a08f27c to VOTER
I20240501 14:10:45.258955 6229 consensus_queue.cc:237] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 10, Committed index: 10, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { promote: false } }
I20240501 14:10:45.260232 6353 tablet_copy_client.cc:647] T 420642af520d45ffbc7f2d35f93e75a7 P e9c60449cf3c46f8aca4ab6c535302a4: tablet copy: Starting download of 1 WAL segments...
I20240501 14:10:45.262035 6353 tablet_copy_client.cc:523] T 420642af520d45ffbc7f2d35f93e75a7 P e9c60449cf3c46f8aca4ab6c535302a4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240501 14:10:45.265504 6353 tablet_bootstrap.cc:492] T 420642af520d45ffbc7f2d35f93e75a7 P e9c60449cf3c46f8aca4ab6c535302a4: Bootstrap starting.
I20240501 14:10:45.266254 5254 raft_consensus.cc:1261] T c8a961e711114f369ba06840c062b982 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 LEARNER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:10:45.266916 6044 consensus_queue.cc:1035] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
I20240501 14:10:45.267201 4994 raft_consensus.cc:1261] T c8a961e711114f369ba06840c062b982 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20240501 14:10:45.267699 6229 consensus_queue.cc:1035] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 10, Time since last communication: 0.000s
W20240501 14:10:45.268486 5061 consensus_peers.cc:485] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:45.270584 6044 raft_consensus.cc:2932] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { promote: false } } }
I20240501 14:10:45.271102 5254 raft_consensus.cc:2932] T c8a961e711114f369ba06840c062b982 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { promote: false } } }
I20240501 14:10:45.277967 4994 raft_consensus.cc:2932] T c8a961e711114f369ba06840c062b982 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 10 to 11, 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { promote: false } } }
I20240501 14:10:45.278748 4850 catalog_manager.cc:5484] T c8a961e711114f369ba06840c062b982 P 1c3fdc5272484ac09ef3e5e36a08f27c reported cstate change: config changed from index 10 to 11, 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" committed_config { opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { promote: false } } }
W20240501 14:10:45.282299 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
I20240501 14:10:45.287657 5126 consensus_queue.cc:237] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 1.11, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { promote: false } }
I20240501 14:10:45.289417 5254 raft_consensus.cc:1261] T c8a961e711114f369ba06840c062b982 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:10:45.289469 4994 raft_consensus.cc:1261] T c8a961e711114f369ba06840c062b982 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Refusing update from remote peer e9c60449cf3c46f8aca4ab6c535302a4: Log matching property violated. Preceding OpId in replica: term: 1 index: 11. Preceding OpId from leader: term: 1 index: 12. (index mismatch)
I20240501 14:10:45.289965 6362 consensus_queue.cc:1035] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:10:45.290175 6044 consensus_queue.cc:1035] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:10:45.292069 6044 raft_consensus.cc:2932] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 [term 1 LEADER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { promote: false } } }
I20240501 14:10:45.293134 5254 raft_consensus.cc:2932] T c8a961e711114f369ba06840c062b982 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { promote: false } } }
I20240501 14:10:45.294872 6353 tablet_bootstrap.cc:492] T 420642af520d45ffbc7f2d35f93e75a7 P e9c60449cf3c46f8aca4ab6c535302a4: Bootstrap replayed 1/1 log segments. Stats: ops{read=11 overwritten=0 applied=11 ignored=2} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240501 14:10:45.295210 6353 tablet_bootstrap.cc:492] T 420642af520d45ffbc7f2d35f93e75a7 P e9c60449cf3c46f8aca4ab6c535302a4: Bootstrap complete.
I20240501 14:10:45.295446 6353 ts_tablet_manager.cc:1367] T 420642af520d45ffbc7f2d35f93e75a7 P e9c60449cf3c46f8aca4ab6c535302a4: Time spent bootstrapping tablet: real 0.030s user 0.013s sys 0.015s
I20240501 14:10:45.295804 4839 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c8a961e711114f369ba06840c062b982 with cas_config_opid_index 11: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240501 14:10:45.296190 6353 raft_consensus.cc:348] T 420642af520d45ffbc7f2d35f93e75a7 P e9c60449cf3c46f8aca4ab6c535302a4 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: NON_VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { promote: true } }
I20240501 14:10:45.296396 6353 raft_consensus.cc:729] T 420642af520d45ffbc7f2d35f93e75a7 P e9c60449cf3c46f8aca4ab6c535302a4 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: e9c60449cf3c46f8aca4ab6c535302a4, State: Initialized, Role: LEARNER
I20240501 14:10:45.296554 6353 consensus_queue.cc:260] T 420642af520d45ffbc7f2d35f93e75a7 P e9c60449cf3c46f8aca4ab6c535302a4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 11, Last appended: 2.11, Last appended by leader: 11, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 11 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: NON_VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { promote: true } }
I20240501 14:10:45.297092 6353 ts_tablet_manager.cc:1398] T 420642af520d45ffbc7f2d35f93e75a7 P e9c60449cf3c46f8aca4ab6c535302a4: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240501 14:10:45.297641 5276 tablet_copy_service.cc:342] P 1c3fdc5272484ac09ef3e5e36a08f27c: Request end of tablet copy session e9c60449cf3c46f8aca4ab6c535302a4-420642af520d45ffbc7f2d35f93e75a7 received from {username='slave'} at 127.29.124.130:54039
I20240501 14:10:45.297756 5276 tablet_copy_service.cc:433] P 1c3fdc5272484ac09ef3e5e36a08f27c: ending tablet copy session e9c60449cf3c46f8aca4ab6c535302a4-420642af520d45ffbc7f2d35f93e75a7 on tablet 420642af520d45ffbc7f2d35f93e75a7 with peer e9c60449cf3c46f8aca4ab6c535302a4
I20240501 14:10:45.297505 4850 catalog_manager.cc:5484] T c8a961e711114f369ba06840c062b982 P e9c60449cf3c46f8aca4ab6c535302a4 reported cstate change: config changed from index 11 to 12, VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) evicted. New cstate: current_term: 1 leader_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240501 14:10:45.297483 4994 raft_consensus.cc:2932] T c8a961e711114f369ba06840c062b982 P 04088ee3bea440a7bd5ff772152b9410 [term 1 FOLLOWER]: Committing config change with OpId 1.12: config changed from index 11 to 12, VOTER 938a24b0e6a040b7966225af26558ff8 (127.29.124.133) evicted. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { promote: false } } }
W20240501 14:10:45.318617 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:10:45.354948 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:10:45.476083 5172 maintenance_manager.cc:419] P e9c60449cf3c46f8aca4ab6c535302a4: Scheduling UndoDeltaBlockGCOp(420642af520d45ffbc7f2d35f93e75a7): 4579 bytes on disk
I20240501 14:10:45.476656 5063 maintenance_manager.cc:643] P e9c60449cf3c46f8aca4ab6c535302a4: UndoDeltaBlockGCOp(420642af520d45ffbc7f2d35f93e75a7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4}
W20240501 14:10:45.504210 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:45.514660 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:45.555882 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:45.568825 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:45.641372 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:45.649098 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:45.649551 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:10:45.688701 5126 raft_consensus.cc:1203] T 420642af520d45ffbc7f2d35f93e75a7 P e9c60449cf3c46f8aca4ab6c535302a4 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.10->[2.11-2.11] Dedup: 2.11->[]
W20240501 14:10:45.691660 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:45.711692 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:45.720948 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:10:45.730186 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:10:45.782428 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:45.788494 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:45.796209 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:45.817219 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:45.842438 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:45.888305 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:10:45.942225 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:45.957775 4840 catalog_manager.cc:4573] TS 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9d32e0707d244f2c9bbcb57f62b09f31: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:45.968637 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:45.970058 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:45.982002 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:46.020634 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:46.086056 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240501 14:10:46.092150 6385 raft_consensus.cc:1050] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c: attempting to promote NON_VOTER e9c60449cf3c46f8aca4ab6c535302a4 to VOTER
I20240501 14:10:46.092831 6385 consensus_queue.cc:237] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 11, Committed index: 11, Last appended: 2.11, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { promote: false } }
I20240501 14:10:46.095131 5126 raft_consensus.cc:1261] T 420642af520d45ffbc7f2d35f93e75a7 P e9c60449cf3c46f8aca4ab6c535302a4 [term 2 LEARNER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 14:10:46.095304 5384 raft_consensus.cc:1261] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Refusing update from remote peer 1c3fdc5272484ac09ef3e5e36a08f27c: Log matching property violated. Preceding OpId in replica: term: 2 index: 11. Preceding OpId from leader: term: 2 index: 12. (index mismatch)
I20240501 14:10:46.095860 6049 consensus_queue.cc:1035] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
W20240501 14:10:46.096092 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240501 14:10:46.096123 6385 consensus_queue.cc:1035] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 12, Last known committed idx: 11, Time since last communication: 0.000s
I20240501 14:10:46.099181 6385 raft_consensus.cc:2932] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 LEADER]: Committing config change with OpId 2.12: config changed from index 11 to 12, e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { promote: false } } }
I20240501 14:10:46.099745 5384 raft_consensus.cc:2932] T 420642af520d45ffbc7f2d35f93e75a7 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { promote: false } } }
I20240501 14:10:46.102880 5126 raft_consensus.cc:2932] T 420642af520d45ffbc7f2d35f93e75a7 P e9c60449cf3c46f8aca4ab6c535302a4 [term 2 FOLLOWER]: Committing config change with OpId 2.12: config changed from index 11 to 12, e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130) changed from NON_VOTER to VOTER. New config: { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { promote: false } } }
I20240501 14:10:46.103739 4850 catalog_manager.cc:5484] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c reported cstate change: config changed from index 11 to 12, e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" committed_config { opid_index: 12 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240501 14:10:46.130357 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:46.182484 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:46.206284 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:46.256914 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:46.291774 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:10:46.345877 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:10:46.348153 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:46.356387 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:46.404244 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:46.420208 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:46.493923 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:10:46.499524 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:46.635859 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:46.716408 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:46.747730 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:46.767845 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:10:46.769106 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:46.783282 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:46.806211 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:10:46.853969 4840 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 93 similar messages]
W20240501 14:10:46.926770 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:46.963812 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 134 similar messages]
W20240501 14:10:47.003612 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:10:47.005515 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:47.056005 4840 catalog_manager.cc:4573] TS 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 420642af520d45ffbc7f2d35f93e75a7: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:47.107331 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:47.132300 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:47.198240 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:10:47.332989 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:10:47.465759 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 131 similar messages]
W20240501 14:10:47.545285 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:47.566717 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:47.584280 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:10:47.636868 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:47.659978 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:10:47.710219 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:47.733628 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:10:47.795965 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:47.886008 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:47.898618 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:47.948549 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 153 similar messages]
W20240501 14:10:47.949363 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:47.989949 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:48.020445 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:10:48.062438 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:48.084400 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:48.182004 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:10:48.182299 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:48.199865 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:48.212047 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:48.252744 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:48.271400 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:48.278822 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:10:48.284636 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:48.340744 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:48.341194 4840 catalog_manager.cc:4573] TS 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet fee16905387d428aa02cd2070c86c790: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:48.343796 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:48.385138 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:48.451546 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:48.497907 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:10:48.557926 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:48.648450 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:48.663228 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:48.681393 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:48.720798 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240501 14:10:48.758320 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:48.770546 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:48.777542 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:48.779922 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:10:48.782455 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:48.799206 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:10:48.838084 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:48.863804 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:48.886852 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:48.942504 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:48.949095 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:48.959726 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:10:49.118911 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:49.155596 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:49.164497 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:10:49.298501 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:49.330948 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:10:49.345329 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:49.361248 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:49.440228 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:10:49.468322 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20240501 14:10:49.545794 5727 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:49.546018 5727 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:10:49.546785 5727 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:10:49.547137 5254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" is_pre_election: true
I20240501 14:10:49.547341 5254 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 2.
W20240501 14:10:49.548269 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:49.548449 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:10:49.548534 5321 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:10:49.548744 5727 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:10:49.552109 4840 catalog_manager.cc:4573] TS 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c8a961e711114f369ba06840c062b982: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:49.619261 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:49.627113 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:10:49.627509 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:49.713920 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:10:49.876718 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:10:50.021333 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:10:50.036346 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:50.097002 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 14:10:50.097877 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:50.104272 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:10:50.187657 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:50.263533 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:50.278213 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:10:50.347163 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:50.366875 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:50.445374 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:50.475384 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:10:50.503161 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:50.516470 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:50.560559 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:10:50.578934 4840 catalog_manager.cc:4573] TS 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 1a7bbfc15a76450bbc9197037e856e08: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:50.611433 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:50.627012 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:50.630005 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:50.633919 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:50.766304 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:50.821555 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:50.864792 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:10:50.919688 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:50.921284 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:50.952085 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:50.955304 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:10:51.006382 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:51.038714 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:51.039791 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240501 14:10:51.047535 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:10:51.058981 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:51.077088 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:51.127223 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:51.235291 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:51.244259 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:51.256510 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:51.267602 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:51.287464 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:51.292502 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:51.336086 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:10:51.355399 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:10:51.377879 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:51.447839 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:51.454084 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:51.515029 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:10:51.536901 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:51.544348 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:51.558425 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:51.698812 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:51.710631 4840 catalog_manager.cc:4573] TS 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet b284aa82594a44f5885f32fa05844913: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:51.725458 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:51.731905 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:10:51.747040 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:51.856645 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:51.925154 4840 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240501 14:10:51.958667 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:10:51.980799 5322 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 146 similar messages]
W20240501 14:10:52.044122 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:52.047238 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:52.138366 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:10:52.190019 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:52.255697 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
I20240501 14:10:52.391805 5727 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:52.392014 5727 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:10:52.392758 5727 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:10:52.393155 5255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" is_pre_election: true
I20240501 14:10:52.393435 5255 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 2.
W20240501 14:10:52.394559 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:52.394708 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:10:52.394794 5321 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:10:52.395044 5727 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:10:52.462249 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:10:52.479502 5192 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 131 similar messages]
W20240501 14:10:52.538426 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:52.558892 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:52.632618 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:10:52.656942 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:10:52.774477 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:52.790532 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:52.938798 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:52.941695 4840 catalog_manager.cc:4573] TS 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 63650547ab444f45bb377bcb49f258ad: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:52.954584 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:52.976914 5062 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 137 similar messages]
W20240501 14:10:52.998929 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:53.007974 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:53.040377 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:10:53.074426 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:53.075107 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:53.084821 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:53.087013 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:53.164582 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:10:53.260862 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:53.278744 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:10:53.294678 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:53.338445 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:53.379458 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:53.417642 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:53.422843 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:53.491185 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:53.514617 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:53.546265 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:53.598101 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:10:53.629321 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:10:53.663713 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:53.674566 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:53.690804 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:53.728024 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:53.793803 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:53.796926 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:53.819015 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:53.822168 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:10:53.830757 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:53.877552 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:53.937291 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240501 14:10:54.017860 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:54.077507 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:54.094453 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:10:54.105301 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:54.158403 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:54.161300 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:54.201930 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:10:54.251168 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:54.283114 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20240501 14:10:54.299031 6773 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:54.299199 6773 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:10:54.300011 6773 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers be56773105e744919601127a94e98adf (127.29.124.132:35459), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:10:54.300321 5384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf" is_pre_election: true
I20240501 14:10:54.300529 5384 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 2.
W20240501 14:10:54.301328 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:54.301496 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:10:54.301566 5191 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:10:54.301790 6773 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:10:54.361028 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:10:54.374436 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:54.535082 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:54.603351 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:54.611909 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:10:54.628050 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:54.690996 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:10:54.791366 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:54.792912 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:10:54.798079 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:55.137854 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:10:55.224123 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:10:55.237501 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 14:10:55.292425 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:55.306030 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:10:55.334563 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:55.503589 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:55.509501 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:55.527076 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:55.542448 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:55.558451 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:55.605531 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:10:55.614034 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:55.617447 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:10:55.650216 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:55.664562 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:55.674360 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:55.791445 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:55.812985 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:55.819011 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:10:55.824141 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:55.825248 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:55.830546 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:55.848841 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:55.901909 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:55.918175 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:55.926524 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:55.996978 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:10:56.001138 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:10:56.054064 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:56.085431 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:56.086815 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240501 14:10:56.196847 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:56.274515 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:56.297503 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:56.311378 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:56.337639 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:56.338574 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:10:56.390316 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:56.449792 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:56.484786 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:56.495913 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:10:56.502478 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:10:56.582257 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:56.644465 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:56.754454 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:56.801198 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:56.821497 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:56.846505 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:56.884254 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:10:56.894234 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:57.006386 5322 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 138 similar messages]
W20240501 14:10:57.016716 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:10:57.032325 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:57.049938 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:57.094903 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:10:57.343533 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:57.428378 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:10:57.439781 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:57.535954 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 127 similar messages]
W20240501 14:10:57.578954 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:10:57.668092 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:57.673889 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:10:57.821892 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:57.827559 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:57.844687 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:10:57.981400 5062 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240501 14:10:58.047065 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:58.062623 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:58.068747 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:58.100484 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:10:58.124313 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:58.172381 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:58.192662 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:58.195732 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:10:58.212597 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:58.233884 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:58.242892 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:58.275378 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:58.303531 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:10:58.316320 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:58.318734 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:58.323014 4840 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 6 similar messages]
W20240501 14:10:58.324129 4840 catalog_manager.cc:4573] TS 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9d32e0707d244f2c9bbcb57f62b09f31: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:58.367666 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:58.370682 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:58.375236 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:58.448449 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:58.459378 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:58.490958 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:58.540431 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:58.619315 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:58.674880 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:10:58.681120 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:58.686323 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:10:58.713833 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:58.764256 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:10:58.803644 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240501 14:10:58.819916 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:58.852119 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:58.854657 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
I20240501 14:10:58.865548 6945 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:10:58.865706 6945 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:10:58.866380 6945 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers be56773105e744919601127a94e98adf (127.29.124.132:35459), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:10:58.866628 5386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf" is_pre_election: true
I20240501 14:10:58.866804 5386 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 2.
W20240501 14:10:58.867725 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:10:58.867897 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:10:58.867980 5191 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:10:58.868186 6945 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:10:58.901326 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:58.910496 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:59.049911 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:59.064528 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:59.099696 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:59.197052 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:59.210518 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:10:59.213766 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:59.218403 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:10:59.250136 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:10:59.290319 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:59.296454 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:59.542908 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:59.549436 4840 catalog_manager.cc:4573] TS 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 61a274b24f53458f88c2cde4f8dbe4f4: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:10:59.551806 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:59.643487 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:10:59.733103 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:10:59.763690 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:10:59.800674 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:10:59.871618 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:10:59.950336 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:11:00.087831 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:11:00.138119 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:00.224077 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:11:00.237288 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:11:00.311650 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:00.388515 4931 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 14:11:00.456634 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:11:00.462178 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:00.529107 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:11:00.557065 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:00.558219 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:11:00.619226 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:11:00.619351 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:11:00.643136 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:11:00.664263 4840 catalog_manager.cc:4573] TS 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet fee16905387d428aa02cd2070c86c790: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:11:00.666535 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:11:00.693432 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:00.799990 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:11:00.802996 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:11:00.862965 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:11:00.869971 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:11:00.898965 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:11:00.914739 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:11:00.942476 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:11:00.956499 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:01.024248 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:01.026346 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:11:01.042070 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:11:01.069993 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:11:01.071756 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:11:01.075990 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:11:01.080551 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:11:01.143833 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:11:01.189147 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:11:01.207857 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:11:01.215337 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:11:01.240111 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:11:01.270720 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:01.279112 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:11:01.286754 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:11:01.359592 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:11:01.447773 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:01.455920 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240501 14:11:01.463536 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:01.622819 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:11:01.782577 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:11:01.807526 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:11:01.865490 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:01.882938 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:11:01.883275 4840 catalog_manager.cc:4573] TS 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c8a961e711114f369ba06840c062b982: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:11:01.909075 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:11:01.938661 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:01.956346 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:02.009742 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 137 similar messages]
W20240501 14:11:02.010633 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:11:02.085181 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:11:02.166188 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:11:02.176551 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:11:02.223326 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:11:02.337227 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:02.467983 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:02.473542 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:02.528188 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:11:02.537879 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 132 similar messages]
W20240501 14:11:02.772610 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:11:02.790773 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:11:02.834288 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:02.952179 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:11:02.961910 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:02.982800 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:03.001575 5062 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 137 similar messages]
W20240501 14:11:03.058126 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:11:03.079286 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:11:03.106110 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:11:03.135246 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:11:03.146131 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:11:03.166061 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:11:03.234783 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:11:03.284775 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:03.288343 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:11:03.299714 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:11:03.313287 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:11:03.394263 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:11:03.427872 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:11:03.439023 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:03.474581 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:11:03.477429 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:11:03.477625 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:03.490970 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:11:03.501821 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:11:03.522131 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:11:03.606488 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:11:03.646783 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:11:03.653398 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:11:03.663987 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:11:03.684091 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:11:03.731875 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:11:03.738235 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:11:03.750075 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:03.860927 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:03.883524 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:11:03.897662 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:11:03.907016 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:11:03.908969 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:03.920483 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240501 14:11:03.971181 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:11:04.089479 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:11:04.139200 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:11:04.142488 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:11:04.234686 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:04.318995 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:11:04.332059 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:04.374729 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:11:04.425668 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:04.476395 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:11:04.611142 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:11:04.730801 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:04.819801 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:11:04.834117 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:11:04.916334 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:04.968062 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:11:05.038262 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:05.042707 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:11:05.098834 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:11:05.160324 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:05.323154 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:05.397910 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:11:05.416546 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 14:11:05.471666 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:11:05.483527 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:11:05.535131 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:05.567309 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:11:05.574443 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:11:05.595243 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:05.656821 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:11:05.665449 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:11:05.689337 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:11:05.714983 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:05.730002 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:11:05.765429 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:11:05.818291 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:11:05.833561 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:11:05.850041 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:11:05.925665 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:11:05.932117 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:11:05.981911 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:11:05.991779 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:06.014881 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:06.031764 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:11:06.075878 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:11:06.099567 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:06.119580 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:11:06.125357 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:11:06.192665 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:11:06.232399 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:11:06.278587 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:11:06.304458 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:11:06.306890 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:11:06.365331 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:11:06.377424 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:11:06.380600 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:11:06.401574 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:11:06.412947 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:11:06.422048 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240501 14:11:06.506932 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:11:06.596005 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:06.616089 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:06.658879 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:06.693322 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:11:06.836894 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:11:06.900831 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:11:06.917354 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:11:06.987407 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:11:07.059736 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:07.072818 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240501 14:11:07.094745 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:07.165712 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:11:07.203482 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:07.263964 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:11:07.504437 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:11:07.535022 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:11:07.544219 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 128 similar messages]
W20240501 14:11:07.579428 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:07.593415 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:11:07.638964 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:07.652633 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:07.665915 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:11:07.925063 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:11:07.989429 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:11:08.027428 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:11:08.039584 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:08.044379 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240501 14:11:08.045181 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:11:08.047629 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:11:08.135918 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:11:08.138859 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:08.191224 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:11:08.211711 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:08.228739 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:11:08.231349 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:11:08.317552 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
I20240501 14:11:08.324515 7480 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:11:08.324707 7480 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:11:08.325580 7480 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers be56773105e744919601127a94e98adf (127.29.124.132:35459), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:11:08.325935 5384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf" is_pre_election: true
I20240501 14:11:08.326180 5384 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 2.
W20240501 14:11:08.326524 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:11:08.330389 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:11:08.330549 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:11:08.330621 5191 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:11:08.330847 7480 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:11:08.418766 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:11:08.444841 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:11:08.445144 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:11:08.462404 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:11:08.542096 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:11:08.556329 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:08.562980 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:11:08.578508 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:11:08.625722 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:11:08.627142 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:11:08.666925 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:11:08.683858 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:11:08.702178 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:11:08.711360 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:11:08.731563 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:11:08.736229 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:08.757421 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:11:08.797075 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:11:08.832423 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:08.912528 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:11:08.940614 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:11:08.942265 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:11:09.008668 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:11:09.028767 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:11:09.032501 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:09.228996 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240501 14:11:09.241122 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:09.299532 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:11:09.314414 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:09.426762 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:11:09.553802 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:11:09.631270 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:11:09.644443 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:09.655045 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:09.829982 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:11:09.852308 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:09.942512 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:11:09.989984 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:11:10.018571 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:11:10.138485 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:11:10.160826 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:11:10.180003 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:10.255633 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:10.382242 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:10.403307 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:11:10.452060 4931 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 14:11:10.488739 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:11:10.626886 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:11:10.684518 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:11:10.690367 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:11:10.694651 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:11:10.698534 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:11:10.722041 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:10.728471 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:11:10.730835 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:11:10.746834 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:11:10.781554 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:10.783957 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:11:10.807019 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:10.810411 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:11:10.863111 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:11:10.889106 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:11:10.907498 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:11:10.962003 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:11:10.982368 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:11:11.107478 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:11:11.108592 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:11:11.142407 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:11:11.149334 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:11:11.188894 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:11:11.219224 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:11:11.220147 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:11:11.223249 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:11.245441 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:11.246559 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:11:11.253731 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:11.283056 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:11:11.318739 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:11:11.338831 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:11:11.346004 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:11:11.531250 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:11:11.621786 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:11:11.621915 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:11.639946 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:11:11.670782 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:11:11.799746 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:11.806401 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:11.826584 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240501 14:11:11.954813 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:11:12.117206 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240501 14:11:12.130844 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:12.138064 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:11:12.243881 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:12.290369 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:12.420804 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:11:12.449566 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:11:12.534341 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:12.568369 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:11:12.625303 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:11:12.638172 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 129 similar messages]
W20240501 14:11:12.668637 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:11:12.793982 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:12.811228 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:11:12.828085 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:12.910192 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:11:12.933003 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:11:12.943023 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:11:13.020172 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:11:13.049229 5062 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 136 similar messages]
W20240501 14:11:13.063356 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:11:13.080598 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:11:13.104012 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:11:13.147308 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:11:13.147841 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:13.209034 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:13.221639 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:11:13.221900 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:11:13.252154 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:11:13.263338 4839 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: ChangeConfig RPC to 127.29.124.130:34377 timed out after 30.000s (SENT) [suppressed 11 similar messages]
W20240501 14:11:13.265187 4839 catalog_manager.cc:5014] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3d6859d53fd747a5a5e94722626fa522 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER failed with leader e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } }.
Pending config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } }
W20240501 14:11:13.278283 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:11:13.278560 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:11:13.313632 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:13.357694 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:11:13.442638 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:11:13.515089 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:11:13.524775 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:11:13.548118 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:11:13.550560 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:11:13.551024 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:11:13.560256 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:13.592026 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:13.598704 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:11:13.637086 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:11:13.637895 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:11:13.651235 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:11:13.702033 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:11:13.777510 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:11:13.846901 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:13.858259 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:11:13.870746 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:11:13.920367 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:11:13.972815 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:13.987032 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:11:13.990633 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:11:14.058781 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:11:14.093348 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:11:14.169562 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240501 14:11:14.186717 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:14.297434 4839 catalog_manager.cc:5014] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 8cc5eec4600640f7a293877c825f8d96 with cas_config_opid_index 13: ChangeConfig:ADD_PEER:NON_VOTER failed with leader 04088ee3bea440a7bd5ff772152b9410 (127.29.124.129:44033) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 13 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }.
Pending config: opid_index: 14 OBSOLETE_local: false peers { permanent_uuid: "04088ee3bea440a7bd5ff772152b9410" member_type: VOTER last_known_addr { host: "127.29.124.129" port: 44033 } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } }
W20240501 14:11:14.330724 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:14.340579 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:11:14.504967 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:11:14.591908 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:14.607578 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:14.756131 4840 catalog_manager.cc:4573] TS 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9d32e0707d244f2c9bbcb57f62b09f31: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:11:14.772958 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:11:14.774726 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:14.955081 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:11:15.042582 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:11:15.052487 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
I20240501 14:11:15.147220 7896 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:11:15.147428 7896 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:11:15.148416 7896 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers be56773105e744919601127a94e98adf (127.29.124.132:35459), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:11:15.148916 5386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf" is_pre_election: true
I20240501 14:11:15.149154 5386 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 2.
W20240501 14:11:15.150018 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:11:15.150171 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:11:15.150259 5191 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:11:15.150497 7896 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:11:15.155428 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:11:15.158638 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:15.169660 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:11:15.175457 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:11:15.221203 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:15.361858 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:11:15.431866 4839 catalog_manager.cc:5014] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3d6859d53fd747a5a5e94722626fa522 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER failed with leader e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } }.
Pending config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } }
W20240501 14:11:15.447165 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:11:15.450398 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:11:15.470821 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:11:15.471014 4931 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 14:11:15.490401 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:15.534583 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:11:15.552058 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:11:15.602061 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:15.617743 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:11:15.770751 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:11:15.777287 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:15.793758 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:11:15.806208 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:11:15.811942 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:11:15.813125 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:11:15.820127 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:11:15.872905 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:11:15.891337 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:11:15.896632 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:11:15.910054 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:11:15.936453 4840 catalog_manager.cc:4573] TS 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 61a274b24f53458f88c2cde4f8dbe4f4: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:11:15.948220 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:11:15.948674 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:11:15.973838 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:11:15.983574 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:11:16.037202 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:11:16.053319 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:16.082281 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:16.133626 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:11:16.145601 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:11:16.180346 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:11:16.221727 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:11:16.258477 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:11:16.304787 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:11:16.309309 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:11:16.379361 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:11:16.386134 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:16.464704 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:11:16.467128 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240501 14:11:16.485174 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:16.601289 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:11:16.617475 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:11:16.625690 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:11:16.654299 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:16.681412 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:11:16.761699 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:17.005296 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:17.051146 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:17.055402 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:11:17.088692 4840 catalog_manager.cc:4573] TS 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet fee16905387d428aa02cd2070c86c790: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:11:17.162153 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:17.177357 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240501 14:11:17.178155 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:11:17.452148 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:17.459513 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:11:17.481478 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:11:17.495075 4839 catalog_manager.cc:5014] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3d6859d53fd747a5a5e94722626fa522 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER failed with leader e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } }.
Pending config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } }
W20240501 14:11:17.509020 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:17.655035 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 133 similar messages]
W20240501 14:11:17.660879 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:11:17.662444 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:11:17.707055 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:17.717670 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:11:17.726709 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:11:17.839844 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:11:17.841998 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:11:17.888589 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:11:17.912627 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:11:17.977964 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:11:18.004660 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:18.044175 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:18.054394 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240501 14:11:18.104072 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:11:18.184137 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:11:18.186200 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:11:18.208806 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:11:18.209528 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:18.219882 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:11:18.256316 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:11:18.273530 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:11:18.290283 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:11:18.312161 4840 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 15 similar messages]
W20240501 14:11:18.313038 4840 catalog_manager.cc:4573] TS 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c8a961e711114f369ba06840c062b982: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:11:18.321695 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:11:18.345606 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:11:18.364742 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:11:18.442857 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:11:18.448099 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:11:18.505380 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:18.505614 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:11:18.530508 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:11:18.556983 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:11:18.597637 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:18.659267 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:11:18.671088 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:11:18.684886 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:11:18.707135 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:18.793052 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:11:18.811934 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:11:18.879149 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:11:18.921072 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:11:18.924068 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:18.930168 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:11:19.039013 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240501 14:11:19.062768 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:19.148661 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:11:19.220389 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:19.223790 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:11:19.247226 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:11:19.288885 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:11:19.305982 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:11:19.476447 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:19.540023 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:19.568584 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:11:19.674124 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:19.740844 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:11:19.750200 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:11:19.955242 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:19.967989 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:20.074606 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:11:20.133543 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:20.146927 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:11:20.245317 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:11:20.320454 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:11:20.334807 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:11:20.398648 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:11:20.447708 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:11:20.467823 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:11:20.478238 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:11:20.514014 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:20.527017 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:11:20.537969 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:20.550149 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:20.564754 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:11:20.591761 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 14:11:20.592741 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:11:20.601053 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:11:20.638404 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:11:20.705760 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:11:20.754352 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:11:20.759086 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:11:20.792387 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:11:20.805824 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:11:20.924643 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:11:20.962720 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:11:20.974793 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:21.001142 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:11:21.032563 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:21.034603 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:11:21.039134 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:11:21.043876 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:11:21.057539 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:11:21.060631 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:21.150508 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:11:21.151264 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:11:21.176960 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:11:21.346011 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:11:21.355790 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:11:21.400349 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:11:21.409514 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240501 14:11:21.424109 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:11:21.427719 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:21.490502 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:11:21.514782 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:21.589839 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:21.604077 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:11:21.639256 4839 catalog_manager.cc:5014] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3d6859d53fd747a5a5e94722626fa522 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER failed with leader e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } }.
Pending config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } }
W20240501 14:11:21.684353 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:11:21.738168 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:11:21.921708 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:11:21.990052 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:11:22.012264 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:22.038705 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:22.042567 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:22.190465 5322 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240501 14:11:22.252622 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:22.303175 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:11:22.337653 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:11:22.410409 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:22.460494 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:22.500674 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:22.536648 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:11:22.612578 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:11:22.660595 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 131 similar messages]
W20240501 14:11:22.664628 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:11:22.791479 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:22.861063 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:22.889120 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:11:22.911228 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:22.947443 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:22.956260 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:22.990639 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:23.068423 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:23.070291 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 132 similar messages]
W20240501 14:11:23.073885 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:11:23.125552 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:23.143705 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:23.149978 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:23.237205 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:23.251065 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:23.267746 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:23.280138 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:23.287058 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:23.351837 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:23.358119 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:11:23.408064 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:23.454262 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:23.456519 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:23.480327 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:23.480517 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:23.486147 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:23.542187 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:23.562223 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:23.572696 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:23.575065 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:23.635990 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:23.687346 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:23.689836 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240501 14:11:23.706668 8218 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:11:23.706878 8218 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:11:23.707659 8218 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:11:23.708370 5255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" is_pre_election: true
I20240501 14:11:23.708577 5255 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 2.
W20240501 14:11:23.709713 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:11:23.709888 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:11:23.709980 5321 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:11:23.710227 8218 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:11:23.737277 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:23.808329 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:23.900280 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:23.913547 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:23.923986 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240501 14:11:24.006686 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:24.009529 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:24.058225 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:24.144361 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:24.263553 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:24.297375 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:24.440610 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:24.459220 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:24.506372 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:24.510843 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:24.575882 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:24.623759 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:24.674100 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:24.872974 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:24.906191 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:24.965633 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:25.002307 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:25.026070 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:25.027151 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:25.128265 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:25.128873 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:25.260690 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:25.268169 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:25.326964 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:25.349610 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:25.414485 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:25.414921 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:25.448180 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:25.473697 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:25.501170 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:25.545645 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:25.553618 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:25.604367 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:25.636891 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:25.707378 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:25.727511 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:25.788516 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:25.837234 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:25.840682 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:25.867602 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:25.874431 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 14:11:25.875257 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:25.895313 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:25.956027 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:25.964197 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:25.991291 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:26.005055 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:26.053043 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:26.099509 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:26.166826 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:26.187999 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:26.214607 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:26.220567 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:26.237437 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:26.273506 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:26.274585 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:26.365027 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:26.365567 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:26.473485 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:26.553342 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:26.588110 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:26.667904 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:26.692675 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:26.693823 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:26.697759 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240501 14:11:26.806174 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:26.902048 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:26.929016 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:27.102077 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:27.207803 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:27.207856 5322 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240501 14:11:27.214954 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:27.303056 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:27.308421 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:27.385493 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:27.441757 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:27.508761 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:27.553656 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:27.624239 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:27.659762 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:27.662479 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:27.709558 5192 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 127 similar messages]
W20240501 14:11:27.710379 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:27.728483 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:27.786856 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:27.790587 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:27.832324 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:27.866909 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
I20240501 14:11:27.941798 8421 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:11:27.941977 8421 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:11:27.942706 8421 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers be56773105e744919601127a94e98adf (127.29.124.132:35459), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:11:27.943012 5386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf" is_pre_election: true
I20240501 14:11:27.943248 5386 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 2.
W20240501 14:11:27.943982 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:11:27.944147 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:11:27.944240 5191 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:11:27.944484 8421 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:11:27.950132 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:27.981323 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:28.020309 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:28.045101 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:28.062183 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:28.109999 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:28.118909 5062 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 144 similar messages]
W20240501 14:11:28.231732 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:28.258842 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:28.272914 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:28.304518 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:28.346884 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:28.373601 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:28.387008 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:28.390857 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:28.430622 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:28.439908 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:28.442251 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:28.452122 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:28.452158 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:28.559232 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:28.560490 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:28.560556 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:28.561545 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:28.584544 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:28.669332 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:28.672181 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:28.688962 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:28.828896 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:28.833104 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:28.838666 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:28.858633 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:28.955866 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:28.990793 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:29.155521 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:29.187470 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:29.270321 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240501 14:11:29.280496 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:29.297919 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:29.339843 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:29.369130 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:29.383199 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:29.434129 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:29.683903 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:29.748299 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:29.767841 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:29.833441 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:29.877331 4839 catalog_manager.cc:5014] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3d6859d53fd747a5a5e94722626fa522 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER failed with leader e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } }.
Pending config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } }
W20240501 14:11:29.883620 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:29.903806 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:30.020951 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:30.022246 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:30.030865 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:30.035235 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:30.186167 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:30.206851 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:30.207171 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:30.228955 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:30.285636 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:30.292150 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:30.293529 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:30.335508 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:30.340469 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:30.433693 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:30.508591 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:30.552009 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:30.565666 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:30.590513 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:30.728662 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:30.750682 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:30.751253 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:30.761799 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:30.784974 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:30.785023 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:30.817138 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:30.819784 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:30.862617 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:30.891980 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:30.913584 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:30.930306 4931 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 14:11:30.931013 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:30.974884 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:31.002668 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:31.069905 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:31.124670 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:31.134279 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:31.192654 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:31.233366 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:31.257009 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:31.258136 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:31.268951 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:31.269620 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:31.308357 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:31.585520 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:31.586999 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:31.595713 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:31.606891 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:31.690858 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:31.738670 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:31.784818 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:31.848153 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:31.849565 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240501 14:11:31.983129 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:31.991609 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:31.991730 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:32.049190 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:32.159217 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:32.160259 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:32.223904 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 142 similar messages]
W20240501 14:11:32.237715 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:32.316890 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:32.377032 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:32.454267 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:32.565310 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:32.567997 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:32.615881 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:32.640826 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:32.673796 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:32.675283 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:32.722021 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:32.729075 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:32.745306 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240501 14:11:32.746189 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:32.872593 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:32.921065 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:32.946852 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:33.027844 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:33.031061 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:33.047948 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:33.090859 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:33.135447 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 143 similar messages]
W20240501 14:11:33.149860 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:33.170509 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:33.242044 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:33.274958 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:33.293280 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:33.328997 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:33.371312 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:33.383821 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:33.403122 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:33.423627 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:33.441751 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:33.443686 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:33.487768 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:33.577054 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:33.633795 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:33.680814 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:33.705116 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:33.738308 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:33.746528 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:33.761384 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:33.799266 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:33.832877 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:33.848268 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:33.850643 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:33.878645 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:34.051607 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:34.118836 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:34.150388 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:34.182255 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:34.211259 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:34.224903 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:34.280906 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:34.446322 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:34.520627 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:34.565236 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:34.621783 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240501 14:11:34.683516 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:34.733300 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:34.806082 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:34.834758 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:34.835300 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:34.866530 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:34.959461 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:35.024830 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:35.049995 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:35.079836 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:35.111380 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:35.196187 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:35.245528 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:35.270298 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:35.295873 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:35.307891 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:35.332008 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:35.387233 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:35.404027 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:35.530361 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:35.567510 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:35.585562 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:35.669245 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:35.740618 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:35.776368 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:35.897212 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:35.911486 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:35.915931 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:35.918066 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:35.933990 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 14:11:35.934762 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:35.986615 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:36.032060 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:36.058746 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:36.064005 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:36.105010 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:36.125734 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:36.140645 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:36.144280 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
I20240501 14:11:36.148698 8904 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:11:36.148896 8904 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:11:36.149717 8904 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:11:36.150415 5254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" is_pre_election: true
I20240501 14:11:36.150681 5254 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 2.
W20240501 14:11:36.159169 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:11:36.159341 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:11:36.159400 5321 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:11:36.159615 9000 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:11:36.220783 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:36.237110 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:36.256350 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:36.270805 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:36.369735 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:36.426908 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:36.454207 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:36.480299 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:36.491370 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:36.592784 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:36.609223 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:36.737785 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:36.772575 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:36.778990 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:36.845180 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240501 14:11:36.849324 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:36.898936 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:36.924378 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:36.973275 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:36.983214 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:37.146840 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:37.230016 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:37.252097 5322 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 136 similar messages]
W20240501 14:11:37.252960 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:37.269320 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:37.288411 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:37.413151 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:37.417409 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:37.481707 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:37.514410 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:37.584357 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:37.614998 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:37.633833 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:37.701606 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:37.711665 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:37.712076 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:37.722558 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:37.725111 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:37.760555 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240501 14:11:37.821209 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:37.971961 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:37.974756 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:38.063331 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:38.149489 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 143 similar messages]
W20240501 14:11:38.177757 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:38.215253 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:38.244576 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:38.353308 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:38.387333 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:38.397853 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:38.408294 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:38.442467 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:38.449402 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
I20240501 14:11:38.477906 9086 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:11:38.478117 9086 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:11:38.479032 9086 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers be56773105e744919601127a94e98adf (127.29.124.132:35459), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:11:38.479327 5386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf" is_pre_election: true
I20240501 14:11:38.479549 5386 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 2.
W20240501 14:11:38.480876 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:11:38.481060 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:11:38.481165 5191 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:11:38.481418 9086 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:11:38.520430 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:38.543642 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:38.560453 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:38.589799 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:38.608299 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:38.652963 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:38.696831 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:38.710777 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:38.726465 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:38.729187 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:38.841234 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:38.860180 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:38.915352 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:38.930624 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:39.012018 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:39.027232 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:39.059439 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:39.079572 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:39.086999 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:39.225364 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:39.278383 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:39.306208 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:39.312726 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:39.416738 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240501 14:11:39.426908 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:39.485705 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:39.501279 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:39.523608 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:39.549062 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:39.704715 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:39.730798 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:39.842623 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:39.899503 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:39.933130 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:39.952281 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:40.011023 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:40.014230 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:40.061842 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:40.067373 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:40.127637 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:40.135426 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:40.163774 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:40.192849 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:40.290287 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:40.322072 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:40.331377 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:40.466710 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:40.585992 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:40.685756 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:40.688237 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:40.770830 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:40.799834 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:40.868913 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:40.871618 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:40.950874 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:40.961652 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:40.970816 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:40.971091 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:40.996137 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 14:11:40.996979 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:41.016597 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:41.055706 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:41.068945 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:41.070039 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:41.071122 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:41.252327 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:41.311347 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:41.325196 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:41.358198 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:41.417451 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:41.445227 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:41.486033 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:41.522917 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:41.530421 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:41.562076 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:41.570010 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:41.587453 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:41.595577 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:41.628353 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:41.758961 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:41.774222 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:41.872980 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:41.917704 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:41.922288 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:41.980122 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:41.987120 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:42.024601 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240501 14:11:42.027570 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:42.064188 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:42.127236 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:42.220208 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:42.306629 5322 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240501 14:11:42.315119 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:42.382263 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:42.416154 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:42.478595 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:42.501883 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:42.512477 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:42.562626 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:42.571696 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:42.571995 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:42.612078 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:42.703642 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:42.765470 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:42.780926 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240501 14:11:42.781759 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:42.816293 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:42.892017 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:42.998148 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:43.000072 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:43.155920 5062 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 133 similar messages]
W20240501 14:11:43.171952 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:43.279788 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:43.297777 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:43.308760 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:43.310029 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:43.312661 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:43.391774 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:43.467744 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:43.473317 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:43.486160 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:43.538952 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:43.560626 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:43.592658 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:43.594214 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:43.616299 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:43.728657 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:43.790424 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:43.821942 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:43.834932 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:43.865949 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:43.931079 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:43.991591 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:44.020565 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:44.023649 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:44.112563 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:44.171926 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:44.233428 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:44.244489 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:44.269758 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:44.288741 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:44.291954 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:44.345360 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:44.391176 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240501 14:11:44.402441 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:44.435035 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:44.452800 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:44.555124 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:44.630028 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:44.650444 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:44.694187 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:44.843540 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:44.867648 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:44.894779 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:44.972880 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:44.982636 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:45.012308 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:45.024791 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:45.053750 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:45.080433 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:45.087682 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:45.134766 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:45.149775 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:45.342808 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:45.456930 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:45.521512 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:45.528780 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:45.592326 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:45.645622 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:45.687964 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:45.761041 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:45.764573 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:45.860648 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:45.874027 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:45.914739 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:45.918538 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:45.937925 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:45.989070 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:46.031590 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:46.090477 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 14:11:46.091266 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:46.091324 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:46.103432 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:46.127418 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:46.187636 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:46.215189 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:46.267606 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:46.285758 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:46.299861 4839 catalog_manager.cc:5014] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 3d6859d53fd747a5a5e94722626fa522 with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER failed with leader e9c60449cf3c46f8aca4ab6c535302a4 (127.29.124.130:34377) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } }.
Pending config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "e9c60449cf3c46f8aca4ab6c535302a4" member_type: VOTER last_known_addr { host: "127.29.124.130" port: 34377 } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: false } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: NON_VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: true } }
I20240501 14:11:46.373301 9346 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:11:46.373517 9346 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:11:46.374354 9346 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers be56773105e744919601127a94e98adf (127.29.124.132:35459), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:11:46.374797 5384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf" is_pre_election: true
I20240501 14:11:46.375082 5384 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 2.
W20240501 14:11:46.376201 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:11:46.376376 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:11:46.376473 5191 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:11:46.376740 9346 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:11:46.419356 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:46.425726 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:46.432252 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:46.435958 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:46.496292 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:46.553308 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:46.631206 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:46.664186 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:46.664992 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:46.689733 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:46.700008 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:46.730374 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:46.755043 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:46.887455 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240501 14:11:47.026701 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:47.030939 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:47.072481 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:47.092622 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:47.101022 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:47.107650 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:47.125756 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:47.126406 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:47.206439 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:47.290897 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:47.322691 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:47.347755 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240501 14:11:47.349951 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:47.408973 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:47.446318 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:47.525189 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:47.529565 4840 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:11:47.530583 4840 catalog_manager.cc:4573] TS 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9d32e0707d244f2c9bbcb57f62b09f31: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:11:47.605343 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:47.615096 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:47.722951 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:47.724155 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:47.769614 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:47.775782 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:47.800067 5192 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 135 similar messages]
W20240501 14:11:47.807790 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:47.981957 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:47.992930 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:48.113747 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:48.154723 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:48.173594 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:48.175652 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:48.218024 5062 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 142 similar messages]
W20240501 14:11:48.222379 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:48.293067 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:48.325510 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:48.390774 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:48.405695 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:48.562182 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:48.564419 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:48.578326 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:48.588284 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:48.609701 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:48.610774 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:48.667784 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:48.684963 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:48.697160 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:48.724810 4840 catalog_manager.cc:4573] TS 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 61a274b24f53458f88c2cde4f8dbe4f4: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:11:48.733768 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:48.839738 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:48.869508 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:48.883124 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:48.890630 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:48.914510 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:49.092602 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:49.140444 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:49.210425 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:49.220396 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:49.232326 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:49.249629 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:49.273160 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:49.308229 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:49.332919 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:49.431061 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240501 14:11:49.448495 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:49.469509 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:49.495234 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:49.496232 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:49.628212 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:49.673509 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:49.773572 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:49.781456 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:49.791227 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:49.798278 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:49.876533 4840 catalog_manager.cc:4573] TS 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet fee16905387d428aa02cd2070c86c790: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:11:49.895973 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:49.917011 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:49.943898 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:49.944729 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:50.051190 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:50.091091 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:50.116474 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:50.161168 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:50.195725 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:50.357285 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:50.398525 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:50.468696 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:50.524149 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:50.534593 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:50.537607 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:50.639724 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:50.640998 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:50.733562 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:50.869168 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:50.876855 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:50.928236 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:50.975217 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:50.995956 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:50.996613 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:51.036135 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:51.108106 4840 catalog_manager.cc:4573] TS 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c8a961e711114f369ba06840c062b982: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:11:51.111371 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:51.123564 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:51.149029 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 14:11:51.166080 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:51.252723 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:51.279711 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:51.281927 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:51.282122 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:51.299559 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:51.350224 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:51.351107 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:51.374538 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:51.413010 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:51.471107 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:51.541450 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:51.613152 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:51.757978 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:51.779786 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:51.783125 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:51.794430 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:51.809892 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:51.849110 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:51.945501 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:52.003295 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:52.015969 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:52.017485 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:52.020562 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240501 14:11:52.021348 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:52.067421 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:52.170876 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:52.185317 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:52.259773 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240501 14:11:52.263392 9825 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:11:52.263545 9825 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:11:52.264217 9825 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers be56773105e744919601127a94e98adf (127.29.124.132:35459), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:11:52.264492 5384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf" is_pre_election: true
I20240501 14:11:52.264743 5384 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 2.
W20240501 14:11:52.265600 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:11:52.265722 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:11:52.265774 5191 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:11:52.265951 9825 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:11:52.379106 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:52.385464 5322 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 135 similar messages]
W20240501 14:11:52.415457 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:52.419579 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:52.440470 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:52.464350 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:52.581192 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:52.598425 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:52.617450 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:52.643213 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:52.751847 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:52.811475 5192 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240501 14:11:52.815436 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:52.852886 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:52.892297 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:52.901604 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:52.981597 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:53.102257 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:53.137931 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:53.139153 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:53.231371 5062 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240501 14:11:53.281941 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:53.385926 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:53.416103 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:53.463469 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:53.494307 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:53.496032 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:53.510646 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:53.519143 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:53.552472 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:53.597249 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:53.656059 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:53.660621 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:53.759852 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240501 14:11:53.776455 9878 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:11:53.776645 9878 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:11:53.777616 9878 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:11:53.777951 5254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" is_pre_election: true
I20240501 14:11:53.778155 5254 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 2.
W20240501 14:11:53.779039 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:11:53.779232 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:11:53.779345 5321 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:11:53.779589 9878 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:11:53.790261 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:53.840937 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:53.850236 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:53.882624 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:53.898564 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:53.900388 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:53.938519 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:53.947672 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:54.021606 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:54.120317 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:54.182408 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:54.220634 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:54.258308 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:54.300184 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:54.333442 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:54.370952 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:54.385360 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:54.421470 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:54.421933 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240501 14:11:54.545766 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:54.548700 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:54.551851 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:54.595960 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:54.599996 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:54.658825 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:54.709111 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:54.750619 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:54.780905 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:54.781316 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:54.839337 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:54.917090 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:55.072214 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:55.141289 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:55.146987 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:55.195967 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:55.241788 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:55.254704 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:55.283532 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:55.377419 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:55.478582 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:55.497565 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:55.528831 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:55.682678 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:55.722579 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:55.799762 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:55.835141 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:55.866233 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:55.892897 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:55.941062 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:55.968608 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:55.990921 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:56.031026 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:56.037789 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:56.042786 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:56.051478 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:56.065011 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:56.100656 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:56.174973 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:56.236572 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:56.266505 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:56.289590 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:56.348301 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240501 14:11:56.349071 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:56.443187 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:56.449153 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:56.449756 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:56.530514 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:56.646258 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:56.673111 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:56.705137 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:56.706869 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:56.709897 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:56.805725 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:56.814754 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240501 14:11:56.815014 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:11:56.855331 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:56.922992 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:56.923696 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:11:56.933190 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:56.965384 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:57.020269 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:57.040458 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:57.053282 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:57.207183 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:57.220088 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:11:57.276701 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:57.280124 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:57.285151 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:57.306782 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:57.419512 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 146 similar messages]
W20240501 14:11:57.454243 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:57.544643 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:57.592432 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:11:57.674633 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:57.778556 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:11:57.830214 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240501 14:11:57.831033 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:11:57.842520 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:11:57.845345 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:11:57.853756 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:57.903623 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:58.040023 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:11:58.110882 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:58.176883 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:58.184232 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:11:58.251818 5062 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240501 14:11:58.263633 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:58.348788 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:11:58.371042 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:58.406620 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:11:58.430747 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:58.486512 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:11:58.499426 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:11:58.514159 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:11:58.529599 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:11:58.567418 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:11:58.588124 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:11:58.595539 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:11:58.606377 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:11:58.609481 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:58.648937 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:11:58.742134 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:11:58.747428 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:58.755803 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:11:58.888929 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:11:58.979689 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:58.990870 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:11:59.126165 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:59.190395 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:11:59.214584 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:59.320819 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:11:59.324127 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:11:59.329336 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:11:59.339685 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:11:59.373903 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:59.408365 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240501 14:11:59.440699 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:11:59.466099 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:11:59.488170 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:11:59.507354 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:11:59.546984 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:11:59.558542 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:11:59.566449 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:11:59.571638 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:59.588232 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:11:59.653892 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:11:59.693961 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:11:59.743924 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:11:59.751397 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:11:59.824899 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:11:59.824909 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:12:00.009900 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:12:00.076826 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:00.223930 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:12:00.322474 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:12:00.347191 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:00.355654 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:12:00.392608 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:12:00.442451 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:00.450474 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:12:00.600917 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:12:00.681943 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:12:00.682269 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:00.762008 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:00.832734 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:12:00.932337 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:12:00.934891 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:12:00.958297 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:00.984680 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:12:01.120956 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:12:01.134174 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:12:01.165987 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:12:01.190102 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:12:01.211774 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:12:01.213294 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:12:01.235673 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:01.257578 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:12:01.268874 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:12:01.277941 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:01.295531 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:12:01.296489 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:12:01.369400 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:12:01.450065 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 14:12:01.468734 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:01.502339 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:12:01.571105 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:12:01.659037 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240501 14:12:01.743093 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
I20240501 14:12:01.805122 10251 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:12:01.805361 10251 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:12:01.806247 10251 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers be56773105e744919601127a94e98adf (127.29.124.132:35459), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:12:01.806744 5384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf" is_pre_election: true
I20240501 14:12:01.807029 5384 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 2.
W20240501 14:12:01.807703 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:12:01.807873 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:12:01.807966 5191 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:12:01.808229 10251 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:12:01.811386 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:12:01.844297 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:01.845103 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:01.851258 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:12:01.851919 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:12:01.852294 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:12:01.898208 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:12:01.912484 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:01.922726 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:12:01.926143 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:12:01.963863 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:12:01.966365 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:12:01.991403 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:12:02.019685 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:12:02.024240 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:12:02.124990 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:12:02.295857 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:02.369211 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:12:02.399029 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:02.412189 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:12:02.417296 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:02.421340 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:12:02.471505 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240501 14:12:02.490721 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:12:02.650571 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:12:02.687892 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:02.771314 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:12:02.824048 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:02.883646 5192 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 133 similar messages]
W20240501 14:12:02.884474 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:02.903550 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:12:03.045228 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:12:03.081684 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:12:03.187425 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:03.190841 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:12:03.216634 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:12:03.296366 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 138 similar messages]
W20240501 14:12:03.325531 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:03.328570 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:12:03.408288 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:03.486187 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:12:03.545868 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:12:03.549021 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:12:03.550261 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:12:03.552752 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:12:03.618894 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:12:03.686096 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:12:03.694993 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:12:03.741048 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:12:03.772198 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:12:03.776890 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:03.814380 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:12:03.817580 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:12:03.896915 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:12:03.944201 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:03.958451 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:12:03.965516 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:03.988590 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:12:04.045464 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240501 14:12:04.053674 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:12:04.062688 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:12:04.213846 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:12:04.292263 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:12:04.299247 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:12:04.337958 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:12:04.392529 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:04.436026 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:04.461079 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:04.509658 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:12:04.521656 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:12:04.530733 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:12:04.542898 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:12:04.571092 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:12:04.622116 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:12:04.663620 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:12:04.668552 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:12:04.799620 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:12:04.807090 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:12:04.817589 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:12:04.825562 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:04.904618 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:04.939028 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:04.991561 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:12:05.071635 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:12:05.238045 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:12:05.247440 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:12:05.323866 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:05.355083 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:05.411571 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:05.423702 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:12:05.553480 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:12:05.595017 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:12:05.625607 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:12:05.646733 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:12:05.723337 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:12:05.752988 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:12:05.796378 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:05.804091 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:05.855677 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:12:05.882028 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:05.933408 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:12:06.105720 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:12:06.125003 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:12:06.249130 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:12:06.276647 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:12:06.321007 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:12:06.330055 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240501 14:12:06.363107 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:12:06.382472 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:06.395874 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:12:06.397459 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:06.410806 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:06.457000 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:12:06.481828 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 14:12:06.482658 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:12:06.519335 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:12:06.551080 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:12:06.596486 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:12:06.600535 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:12:06.638445 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:12:06.768474 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:06.802469 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:12:06.881909 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:06.926970 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:12:06.935989 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:12:06.985008 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:12:06.991513 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:07.016677 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:12:07.017391 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:12:07.017489 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:12:07.033645 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:12:07.068621 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:12:07.165632 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:07.172686 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:12:07.189725 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:12:07.288149 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:12:07.291283 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:12:07.324405 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:12:07.396171 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:07.438450 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:12:07.480801 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:12:07.497748 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240501 14:12:07.533676 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:12:07.586448 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:07.669155 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:12:07.678848 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:07.747722 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:12:07.871203 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:12:07.962374 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:08.003897 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:12:08.031980 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 127 similar messages]
W20240501 14:12:08.138607 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:08.170310 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:08.220504 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:12:08.228602 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:12:08.262828 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:12:08.266535 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:12:08.308059 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:12:08.337469 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 138 similar messages]
W20240501 14:12:08.338364 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:12:08.447202 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:08.506492 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:12:08.510604 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:12:08.577315 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:08.642691 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:12:08.649078 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:12:08.670607 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:08.708375 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:12:08.734870 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:12:08.826331 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:12:08.836424 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:12:08.869176 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240501 14:12:08.916576 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:08.960894 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:12:08.962047 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:09.077880 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:12:09.098179 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:12:09.129815 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:12:09.139012 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:12:09.187444 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:12:09.226540 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:12:09.266043 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:09.289096 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:12:09.391353 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:12:09.409870 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:12:09.427119 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:12:09.462714 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:12:09.462839 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:09.499512 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:12:09.504634 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:09.535143 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:12:09.537973 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:12:09.650645 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240501 14:12:09.680596 10598 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:12:09.680825 10598 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:12:09.681779 10598 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:12:09.682118 5254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" is_pre_election: true
I20240501 14:12:09.682327 5254 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 2.
W20240501 14:12:09.683152 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:12:09.683367 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:12:09.683458 5321 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:12:09.683686 10598 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:12:09.691421 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:12:09.733697 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:12:09.759721 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:12:09.879988 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:12:09.885351 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:09.963243 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:09.963349 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:12:09.969985 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:12:10.016734 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:12:10.081765 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:10.163760 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:12:10.181298 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:12:10.188406 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:12:10.278322 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:10.394233 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:12:10.431761 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:10.481526 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:12:10.488430 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:10.581990 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:12:10.599160 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:12:10.650525 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:12:10.717559 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:12:10.749804 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:12:10.762600 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:12:10.809074 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:10.826567 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:10.924508 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:10.989024 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:12:11.030619 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:12:11.056674 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:12:11.118903 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:12:11.119339 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:12:11.141417 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:12:11.147856 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240501 14:12:11.310433 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:12:11.313350 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:12:11.368427 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:11.410377 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:12:11.427718 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:11.436758 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:12:11.526654 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:11.559828 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 14:12:11.611467 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:12:11.643967 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:12:11.779510 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:12:11.807916 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:12:11.826505 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:12:11.845350 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:12:11.885851 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:12:11.903121 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:12:11.914981 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:11.940462 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:12:11.942382 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:11.952045 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:11.962394 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:12:12.013990 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:12:12.075330 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:12:12.132393 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:12:12.211752 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:12:12.224628 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:12:12.240564 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:12:12.264032 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:12:12.305622 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:12:12.366434 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:12:12.427130 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:12.454056 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:12.485311 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:12:12.502333 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 143 similar messages]
W20240501 14:12:12.542598 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:12:12.575322 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:12.615818 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:12:12.684343 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:12:12.726143 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:12:12.836757 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:12:12.839524 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:12.906505 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:12:12.978431 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:12:12.983966 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:12:13.006692 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:13.059190 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 129 similar messages]
W20240501 14:12:13.059891 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:13.101073 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:12:13.170563 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:12:13.276597 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:12:13.283083 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:12:13.338642 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 137 similar messages]
W20240501 14:12:13.442270 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:13.503062 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:12:13.517971 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:13.529629 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:12:13.564502 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:12:13.630630 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:13.635371 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:12:13.640429 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:12:13.646631 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:12:13.680066 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:12:13.680806 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:12:13.856235 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:12:13.901903 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:13.914156 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:12:13.981246 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:12:14.004338 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240501 14:12:14.099642 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:14.112756 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:12:14.123292 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:14.186414 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:12:14.187417 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:12:14.220033 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:12:14.261504 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:12:14.269033 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:12:14.427735 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:14.442277 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:12:14.474718 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:12:14.475704 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:12:14.488700 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:14.505514 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:12:14.524070 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:12:14.541549 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:14.572667 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:12:14.753921 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:12:14.825476 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:12:14.826092 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:12:14.856851 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:12:14.944746 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:12:14.949069 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:12:14.978940 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:14.978971 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:12:15.007879 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:15.023712 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:12:15.093537 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:12:15.122072 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:15.350952 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:12:15.380606 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:15.387627 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:15.410085 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:12:15.506558 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:12:15.525830 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:15.555377 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:12:15.608538 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:12:15.690742 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:12:15.699251 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:12:15.749521 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:12:15.818421 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:15.820484 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:12:15.941525 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:15.947212 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:16.047142 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:12:16.069049 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:12:16.088271 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:12:16.132267 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:12:16.142961 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:12:16.160648 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:12:16.189989 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:12:16.257185 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:12:16.307060 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:12:16.350750 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:16.370177 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:12:16.413973 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:12:16.418228 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:16.420356 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:12:16.455566 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:16.518025 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240501 14:12:16.530352 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:12:16.547447 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:12:16.565003 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:12:16.579882 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:12:16.649191 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 14:12:16.737676 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:12:16.814298 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:12:16.816848 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:16.822432 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:12:17.027303 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:17.036928 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:17.055330 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:12:17.069630 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:12:17.211866 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:12:17.257887 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:12:17.279244 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:12:17.301029 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:12:17.324687 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:12:17.326912 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:12:17.354331 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:12:17.370795 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:17.375831 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:12:17.437714 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:12:17.444217 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:17.533284 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 142 similar messages]
W20240501 14:12:17.534137 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:12:17.587447 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:17.587451 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:12:17.832998 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:12:17.844997 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:12:17.846417 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:17.883489 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:17.886731 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:12:17.887511 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:12:17.893632 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:12:17.991371 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:12:18.059417 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:12:18.085083 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 131 similar messages]
W20240501 14:12:18.176502 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:18.229508 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:12:18.305989 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:18.346633 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:12:18.365581 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:18.385746 5062 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 137 similar messages]
W20240501 14:12:18.388449 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:12:18.400961 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:12:18.430460 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:12:18.471247 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:12:18.595885 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:12:18.626518 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:18.627173 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:12:18.718612 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:12:18.773319 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:12:18.808979 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:18.822880 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:12:18.852564 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:12:18.889991 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:12:18.893579 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:18.912042 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:12:18.934657 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:12:18.974171 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:12:19.001538 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:12:19.034534 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240501 14:12:19.084992 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:19.092268 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:12:19.100818 4838 catalog_manager.cc:5014] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet de6c7753c8564107b858876fcfa5924d with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER failed with leader 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }.
Pending config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } }
W20240501 14:12:19.164667 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:12:19.221455 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:12:19.309934 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:12:19.324052 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:12:19.375211 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:19.379056 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:19.410661 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:12:19.447639 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:12:19.531001 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:12:19.626091 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:19.676429 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:12:19.679431 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:12:19.798415 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:19.847455 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:12:19.859848 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:12:19.893457 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:12:19.896520 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:12:19.908622 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:19.947420 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:12:20.036249 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:12:20.044409 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:20.131570 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:12:20.194790 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:12:20.251597 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:12:20.256937 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:12:20.322779 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:20.366258 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:12:20.388442 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240501 14:12:20.389498 11222 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:12:20.389652 11222 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:12:20.390322 11222 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:12:20.390726 5254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" is_pre_election: true
I20240501 14:12:20.390961 5254 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 2.
W20240501 14:12:20.391806 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:12:20.391980 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:12:20.392051 5321 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:12:20.392279 11222 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:12:20.415767 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:12:20.459342 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:20.465353 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:20.643510 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:12:20.706902 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:12:20.815963 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:20.821133 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:12:20.861075 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:20.932123 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:12:20.938030 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:12:20.942636 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:21.027130 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:12:21.096241 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:12:21.157851 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:12:21.181795 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:12:21.259516 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:12:21.278548 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:12:21.313143 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:12:21.324566 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:21.385562 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:12:21.412532 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:12:21.429594 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:21.442808 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:21.442831 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:12:21.445835 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:12:21.492995 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:12:21.501473 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:12:21.524606 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:12:21.532123 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240501 14:12:21.648401 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:12:21.681200 4931 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 14:12:21.682036 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:12:21.720170 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:12:21.796509 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:12:21.809085 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:12:21.849361 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:21.853178 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:21.867704 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:12:21.932616 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:12:22.014098 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:22.037225 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:12:22.164470 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:12:22.173674 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:12:22.238493 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:22.249270 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:12:22.284312 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:22.321453 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:12:22.451375 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:12:22.470754 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:22.472657 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:12:22.498147 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:12:22.605891 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:12:22.639914 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 142 similar messages]
W20240501 14:12:22.640949 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:12:22.698997 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:22.762831 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:12:22.788697 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:12:22.819840 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:22.856645 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:12:22.864642 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:12:22.949834 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:22.987573 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:12:23.032685 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:12:23.073067 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:12:23.130210 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 126 similar messages]
W20240501 14:12:23.178174 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:12:23.241500 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:23.243897 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:12:23.282732 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:12:23.308872 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:12:23.352399 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:23.407061 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:23.407892 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:12:23.422744 5062 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 136 similar messages]
W20240501 14:12:23.433440 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:12:23.696831 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:12:23.706204 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:12:23.725627 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:12:23.778872 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:12:23.859486 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:12:23.859596 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:23.860702 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:12:23.889039 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:23.924808 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:23.944557 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:12:23.956229 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240501 14:12:23.969336 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:12:23.972935 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:12:23.990882 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:12:24.058481 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:12:24.075203 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:12:24.139207 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:12:24.149070 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:12:24.245532 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:12:24.252100 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:12:24.330762 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:24.371127 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:12:24.434130 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:24.537722 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:24.584846 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:12:24.604210 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:12:24.618443 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:12:24.620177 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:12:24.659170 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:12:24.704310 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:12:24.718261 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:24.758689 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:12:24.780428 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:12:24.852392 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:24.928139 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:12:24.954113 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:12:24.960245 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:24.981498 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:12:25.122597 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:25.149843 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:12:25.158331 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:12:25.256254 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:12:25.256737 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:12:25.334955 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:25.362341 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:12:25.507552 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:12:25.524338 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:12:25.563396 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:25.579538 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:25.597715 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:12:25.701288 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:12:25.811937 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:25.839587 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:12:25.862428 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:12:25.867704 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:12:25.882045 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:12:25.949842 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:12:26.004807 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:26.011071 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:26.042460 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:12:26.137509 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:12:26.196753 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:12:26.219568 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:26.245494 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:12:26.366619 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:12:26.408056 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:12:26.410704 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:12:26.459569 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:26.474205 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:12:26.485584 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:12:26.492455 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:12:26.493422 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240501 14:12:26.502808 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:12:26.516651 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:12:26.563155 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:26.587996 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:12:26.671454 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:12:26.701747 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 14:12:26.711483 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:26.723793 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:12:26.754494 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:12:26.859627 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:12:26.949061 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:12:26.951890 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:12:26.974432 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:12:27.029982 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:27.062862 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:12:27.152160 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:12:27.162858 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:27.224795 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:12:27.240974 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:12:27.241541 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:12:27.244760 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:27.400424 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:12:27.509771 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:27.551312 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:12:27.580224 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:12:27.592864 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:12:27.629017 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:12:27.642340 5322 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 144 similar messages]
W20240501 14:12:27.643054 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:27.666440 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:12:27.685467 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:12:27.709092 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:27.783064 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:12:27.800017 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:12:27.834956 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240501 14:12:27.868738 11652 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:12:27.868939 11652 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:12:27.869837 11652 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:12:27.870265 5254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" is_pre_election: true
I20240501 14:12:27.870465 5254 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 2.
W20240501 14:12:27.871258 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:12:27.871429 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:12:27.871510 5321 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:12:27.871752 11652 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:12:27.899933 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:12:27.950464 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:28.023298 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:28.158797 5192 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 135 similar messages]
W20240501 14:12:28.159575 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:28.224864 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:12:28.287508 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:12:28.357805 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:12:28.398485 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:28.412600 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:12:28.450215 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:12:28.467402 5062 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 142 similar messages]
W20240501 14:12:28.468179 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:12:28.515228 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:28.547871 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:12:28.671671 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:12:28.717666 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:28.771540 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:12:28.776520 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:12:28.778908 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:12:28.904718 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:12:28.932262 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:12:28.934671 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:12:28.961989 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:12:28.977450 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:12:28.996448 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:29.008442 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:29.056483 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:12:29.089916 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:12:29.120719 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:12:29.127166 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:12:29.164067 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240501 14:12:29.215018 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:29.260221 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:12:29.340376 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:12:29.373606 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:12:29.404760 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:29.492545 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:12:29.495265 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:29.513461 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:12:29.602509 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:12:29.620337 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:12:29.641088 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:12:29.711170 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:12:29.725009 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:29.729904 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:12:29.787923 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:29.801795 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:12:29.837788 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:12:29.846774 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:12:29.974772 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:29.983829 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:12:30.080240 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:12:30.096390 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:12:30.227458 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:12:30.236289 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:12:30.277341 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:12:30.335566 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:30.341384 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:30.352413 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:12:30.354954 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:12:30.443645 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:12:30.553743 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:30.595280 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:12:30.696025 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:12:30.765673 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:12:30.782889 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:30.793180 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:12:30.800379 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:12:30.884243 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:12:30.892591 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:30.931663 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:31.106441 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:12:31.128590 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:12:31.169732 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:12:31.205971 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:31.349455 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:31.363060 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:12:31.364421 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:12:31.389830 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:12:31.410887 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:12:31.421248 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:12:31.445236 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:12:31.459100 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
I20240501 14:12:31.490936 11816 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:12:31.491134 11816 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:12:31.492100 11816 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers be56773105e744919601127a94e98adf (127.29.124.132:35459), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:12:31.492430 5386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf" is_pre_election: true
I20240501 14:12:31.492617 5386 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 2.
W20240501 14:12:31.493871 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:12:31.494053 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:12:31.494148 5191 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:12:31.494383 11816 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:12:31.516332 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:12:31.613449 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:12:31.623669 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:12:31.648993 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:12:31.665870 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:12:31.705242 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:12:31.793365 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 14:12:31.800544 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:12:31.802707 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:31.817618 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240501 14:12:31.870332 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:12:31.885102 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:12:31.899968 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:32.022251 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:12:32.032732 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:12:32.064056 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:32.098294 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:12:32.151446 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:12:32.171463 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:12:32.183328 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:12:32.208432 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:12:32.338517 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:32.449323 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:12:32.453830 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:12:32.468447 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:12:32.495265 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:12:32.506178 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:12:32.506935 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:32.518085 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:12:32.578070 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:32.595857 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:12:32.699807 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 144 similar messages]
W20240501 14:12:32.740718 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:12:32.827363 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:12:32.926751 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:32.965420 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:32.971482 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:12:33.022698 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:12:33.140285 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:12:33.148916 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:12:33.168368 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:12:33.170313 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 129 similar messages]
W20240501 14:12:33.171047 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:33.312003 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:12:33.358909 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
I20240501 14:12:33.433113 11782 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:12:33.433302 11782 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:12:33.434028 11782 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:12:33.434432 5255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" is_pre_election: true
I20240501 14:12:33.434720 5255 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 2.
W20240501 14:12:33.435510 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:12:33.435678 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:12:33.435742 5321 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:12:33.435935 11782 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:12:33.438932 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:12:33.454546 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:12:33.459921 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:12:33.477942 5062 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240501 14:12:33.553063 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:12:33.559247 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:33.610030 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:12:33.706599 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:33.864025 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:12:33.884802 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:12:33.896057 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:12:33.927176 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:12:33.950395 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:12:33.957726 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:12:33.958526 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:33.984863 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:12:33.987613 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:12:34.022346 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:12:34.071558 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:34.192636 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:12:34.208356 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:12:34.217136 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:12:34.254782 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240501 14:12:34.274757 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:12:34.293125 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:12:34.317170 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:34.378110 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:12:34.446264 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:12:34.499907 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:34.542073 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:12:34.614202 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:12:34.629735 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:12:34.679169 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:34.707548 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:12:34.780092 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:12:34.802410 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:12:34.834761 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:12:34.870194 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:12:34.880210 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:12:34.892591 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:35.004562 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:12:35.054284 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:35.088022 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:12:35.144325 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:12:35.248363 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:12:35.252488 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:35.268743 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:12:35.276947 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:12:35.363150 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:35.387624 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:12:35.431185 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:35.475831 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:12:35.486984 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:12:35.688264 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:35.696259 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:12:35.862838 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:12:35.887307 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:12:35.891316 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:35.898874 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:12:35.918715 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:36.018481 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:12:36.024175 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:12:36.115011 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:12:36.147068 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:12:36.189512 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:12:36.204483 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:36.215071 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:12:36.286414 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:12:36.295601 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:12:36.408614 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:12:36.444630 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:12:36.478289 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:12:36.480625 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:36.492785 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:12:36.539471 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:36.632941 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:12:36.683646 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:12:36.685098 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:36.722937 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:12:36.826081 4931 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 14:12:36.826835 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:12:36.830327 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:12:36.929039 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:36.942353 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:36.942869 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:12:36.945168 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:12:37.015022 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:12:37.023903 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:12:37.071198 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:12:37.121275 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240501 14:12:37.152318 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:12:37.213837 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:12:37.219481 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:37.287657 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:12:37.327399 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:12:37.338356 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:12:37.390690 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:12:37.450382 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:12:37.486385 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:37.522445 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:12:37.532668 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:37.565488 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:12:37.584187 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:12:37.600826 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:12:37.750385 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240501 14:12:37.773128 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:12:37.800367 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:37.842103 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:12:37.851635 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:12:37.899792 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:12:37.906160 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:37.987552 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:12:38.037071 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:38.176736 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:38.190261 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 128 similar messages]
W20240501 14:12:38.325943 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:12:38.329571 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:12:38.389703 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:12:38.442257 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:12:38.449327 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:38.469048 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:38.496971 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 143 similar messages]
W20240501 14:12:38.498111 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:12:38.521076 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:12:38.585616 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:12:38.608703 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:12:38.608944 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:12:38.621178 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:38.680863 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:12:38.734604 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:12:38.756485 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:12:38.825469 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:12:38.855240 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:38.867866 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:12:39.065200 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:39.113099 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:12:39.173626 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:12:39.190431 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:39.238835 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:12:39.277281 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:39.280694 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:12:39.299216 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:12:39.406313 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:12:39.412022 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:12:39.524714 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:39.531622 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:12:39.535780 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:12:39.541234 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:12:39.548461 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:12:39.562114 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:12:39.582882 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:39.685775 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:12:39.690012 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:12:39.728502 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:39.789383 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:12:39.866380 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:12:39.885156 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240501 14:12:39.901840 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:12:39.921131 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:12:39.957163 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:39.977988 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:12:40.006369 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:12:40.018529 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:12:40.051137 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:40.104576 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:40.175581 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:12:40.237406 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:12:40.256506 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:12:40.280776 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:12:40.299503 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:12:40.365379 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:12:40.389526 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:40.475780 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:12:40.594882 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:12:40.644618 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:40.656651 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:40.744968 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:12:40.854159 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:12:40.897187 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:12:40.944428 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:12:40.951923 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:12:40.955826 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:40.971473 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:12:41.056648 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:41.071913 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:41.082338 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:12:41.090281 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:12:41.106715 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:12:41.141377 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:12:41.254755 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:12:41.376977 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:12:41.474961 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:12:41.499049 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:12:41.572722 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:41.581681 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:41.629416 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:12:41.656111 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:41.665658 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:12:41.677665 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:12:41.693819 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:12:41.730917 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240501 14:12:41.755153 4843 maintenance_manager.cc:419] P f10c6a9c4bad46deb63190d64169ec11: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=0.033340
I20240501 14:12:41.772964 4842 maintenance_manager.cc:643] P f10c6a9c4bad46deb63190d64169ec11: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.018s user 0.012s sys 0.002s Metrics: {"bytes_written":2662,"delete_count":0,"lbm_write_time_us":167,"lbm_writes_lt_1ms":4,"reinsert_count":0,"thread_start_us":180,"threads_started":1,"update_count":39,"wal-append.queue_time_us":285}
I20240501 14:12:41.773569 4843 maintenance_manager.cc:419] P f10c6a9c4bad46deb63190d64169ec11: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.176795
I20240501 14:12:41.784669 4842 maintenance_manager.cc:643] P f10c6a9c4bad46deb63190d64169ec11: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.011s user 0.010s sys 0.000s Metrics: {"cfile_cache_miss":5,"cfile_cache_miss_bytes":15854,"cfile_init":3,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":457,"dirs.run_cpu_time_us":87,"dirs.run_wall_time_us":88,"lbm_read_time_us":256,"lbm_reads_lt_1ms":17,"lbm_write_time_us":296,"lbm_writes_lt_1ms":9,"reinsert_count":0,"thread_start_us":365,"threads_started":2,"update_count":39}
W20240501 14:12:41.930480 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 14:12:41.930569 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:12:42.021768 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:42.027489 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:12:42.036032 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:12:42.060384 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:12:42.101001 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:12:42.121121 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:12:42.130147 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:12:42.147507 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:42.175832 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:42.199805 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:12:42.250156 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:12:42.266191 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240501 14:12:42.340864 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:12:42.452136 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:12:42.480562 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:12:42.530468 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:12:42.562767 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:12:42.576316 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:42.596822 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:12:42.629858 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:42.638406 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:42.668465 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:12:42.722818 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:12:42.735689 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:12:42.843077 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:12:42.857232 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:12:42.899417 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 143 similar messages]
W20240501 14:12:42.923204 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:12:43.009071 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:12:43.099927 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:43.134037 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:12:43.148511 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:43.189565 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:43.200917 5192 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 129 similar messages]
W20240501 14:12:43.226500 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:12:43.404106 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:12:43.424717 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:12:43.524605 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:43.537000 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:43.554667 5062 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 138 similar messages]
W20240501 14:12:43.558565 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:12:43.578258 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:12:43.618566 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:12:43.697696 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:12:43.712203 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:12:43.739344 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:12:43.752699 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:12:43.763994 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:12:43.765650 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:12:43.806560 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:43.856418 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:12:44.005069 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:44.082639 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:12:44.094380 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:12:44.130455 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:44.143234 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:12:44.184801 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:12:44.316439 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:12:44.331542 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:12:44.379066 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:44.415889 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:12:44.460215 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:12:44.504539 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:12:44.525017 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:12:44.541781 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:44.575202 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:12:44.614886 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:12:44.620427 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:44.811331 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:12:44.840338 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:12:44.864233 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240501 14:12:44.872177 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:12:44.913604 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:12:44.921073 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:44.932911 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:12:45.035123 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:45.043272 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:12:45.111466 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:45.111605 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:12:45.202214 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:12:45.268385 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:12:45.282867 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:12:45.292594 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:12:45.365450 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:12:45.378677 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:12:45.393414 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:45.419600 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:12:45.452908 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:45.461692 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:12:45.596289 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:45.670156 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:12:45.690616 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:12:45.777056 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:45.835130 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:12:45.846081 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:12:45.889062 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:12:45.897697 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:12:45.942075 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:12:45.970233 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:12:46.001245 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:46.066097 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:46.157719 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:12:46.162097 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:12:46.215353 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:12:46.263648 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:46.279827 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:12:46.373390 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:12:46.377014 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:12:46.437691 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:12:46.492211 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:12:46.535140 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:46.548520 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:12:46.553553 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:12:46.564818 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:12:46.627363 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:46.749996 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:12:46.793566 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:12:46.853006 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:12:46.876401 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:46.884235 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:12:46.950934 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:12:47.040763 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 14:12:47.041532 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:12:47.063004 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:12:47.081305 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:12:47.084995 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:47.250550 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:47.301626 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:12:47.393407 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:12:47.404315 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:12:47.471094 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240501 14:12:47.483832 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:47.543535 4840 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 13 similar messages]
W20240501 14:12:47.544548 4840 catalog_manager.cc:4573] TS 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9d32e0707d244f2c9bbcb57f62b09f31: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:12:47.555102 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:12:47.572214 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:12:47.572502 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:12:47.589231 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:47.613387 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:12:47.648562 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:12:47.719645 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:12:47.767467 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:12:47.803351 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:47.899051 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:12:47.937016 5322 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 138 similar messages]
W20240501 14:12:47.949359 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:12:47.979969 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:12:47.990701 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:12:48.001309 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:12:48.107159 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:48.143412 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:12:48.196801 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:12:48.200965 5192 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 127 similar messages]
W20240501 14:12:48.204576 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:48.314641 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:48.387665 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:12:48.429594 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:12:48.494474 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:12:48.500519 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:12:48.527539 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:12:48.566762 5062 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 138 similar messages]
W20240501 14:12:48.575356 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:12:48.617357 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:12:48.630203 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:48.692430 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:48.695271 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:48.701434 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:12:48.705670 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:12:48.733083 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:12:48.740914 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:12:48.755036 4840 catalog_manager.cc:4573] TS 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 61a274b24f53458f88c2cde4f8dbe4f4: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:12:48.864554 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:12:48.940673 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:12:49.031577 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:12:49.084184 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:12:49.149868 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:12:49.159252 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:12:49.180929 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:49.200667 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:12:49.220301 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:49.295050 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:49.299403 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:12:49.380903 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:12:49.411969 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:12:49.418028 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:12:49.435552 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:12:49.449962 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
I20240501 14:12:49.478415 12278 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:12:49.478610 12278 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:12:49.479460 12278 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers be56773105e744919601127a94e98adf (127.29.124.132:35459), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:12:49.479821 5386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf" is_pre_election: true
I20240501 14:12:49.480062 5386 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 2.
W20240501 14:12:49.480831 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:12:49.480975 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:12:49.481065 5191 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:12:49.481287 12278 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:12:49.534337 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:12:49.684693 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:49.720896 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:49.729054 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:12:49.781121 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:49.797410 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:12:49.851433 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:12:49.919312 4840 catalog_manager.cc:4573] TS 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet fee16905387d428aa02cd2070c86c790: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:12:50.015769 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:12:50.043844 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:12:50.054196 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:12:50.082284 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240501 14:12:50.160905 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:12:50.184082 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:50.188005 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:50.197355 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:12:50.215396 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:12:50.325543 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:50.365196 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:12:50.398011 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:12:50.400254 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:12:50.462858 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:12:50.487660 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:12:50.585170 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:50.673753 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:50.736384 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:12:50.794708 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:12:50.862236 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:12:50.913828 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:12:50.943225 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:51.001708 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:12:51.003553 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:12:51.086527 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:12:51.097033 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:12:51.102748 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:51.114939 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:12:51.121181 4840 catalog_manager.cc:4573] TS 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c8a961e711114f369ba06840c062b982: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:12:51.204067 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:12:51.218588 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:12:51.267621 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:51.302170 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:12:51.374737 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:12:51.425187 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:12:51.433755 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
I20240501 14:12:51.525391 12278 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:12:51.525583 12278 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:12:51.526250 12278 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers be56773105e744919601127a94e98adf (127.29.124.132:35459), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:12:51.526510 5386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf" is_pre_election: true
I20240501 14:12:51.526734 5386 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 2.
W20240501 14:12:51.527575 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:12:51.527755 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:12:51.527856 5191 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:12:51.528088 12278 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:12:51.539078 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:12:51.566017 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:51.575491 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:12:51.599207 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:12:51.631713 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:12:51.689862 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:51.732895 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:12:51.753799 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:12:51.763453 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:12:51.796844 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:12:51.864009 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:12:51.866685 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:12:51.890996 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:52.026433 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:12:52.035990 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:12:52.058010 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:52.127400 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 14:12:52.151894 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:12:52.181370 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:12:52.197338 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:52.277905 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:12:52.284291 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:12:52.316390 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:52.475921 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:52.557060 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:12:52.587972 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:12:52.590705 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240501 14:12:52.627676 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:12:52.691634 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:12:52.700309 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:12:52.715674 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:52.801402 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:52.806542 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:12:52.815703 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:12:52.852610 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:12:52.972556 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:12:52.981848 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:12:52.991106 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240501 14:12:53.025043 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:12:53.049970 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:12:53.050151 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:53.240895 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:12:53.243685 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 133 similar messages]
W20240501 14:12:53.273680 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:12:53.276314 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:53.293898 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:53.403808 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:12:53.478817 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:12:53.521822 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:53.577793 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 136 similar messages]
W20240501 14:12:53.578543 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:12:53.603911 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:12:53.610421 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:12:53.631295 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:12:53.743717 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:12:53.747110 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:12:53.750171 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:12:53.765609 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:53.862639 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:53.934619 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:12:53.947055 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:12:53.965219 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:12:54.038630 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:54.042002 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:12:54.113544 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:12:54.171444 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:54.180264 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:12:54.220686 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:12:54.278806 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:12:54.289984 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:12:54.296532 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:54.323477 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:12:54.356506 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:12:54.358968 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:12:54.366340 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:12:54.532809 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:12:54.537781 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:54.540541 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:12:54.556135 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:54.611332 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:12:54.650971 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:12:54.767168 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:12:54.797047 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:54.925637 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:12:54.962610 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240501 14:12:54.986806 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:55.035300 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:55.107797 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:12:55.136404 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:12:55.185067 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:12:55.187124 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:12:55.254323 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:12:55.273275 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:12:55.286650 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:55.307227 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:12:55.317340 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:12:55.356900 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:12:55.472868 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:55.558697 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:12:55.572353 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:12:55.599380 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:12:55.603044 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:55.628930 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:12:55.667007 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:55.738108 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:12:55.969939 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:12:55.977504 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:12:56.034533 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:12:56.041503 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:56.044806 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:12:56.052611 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:56.097121 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:56.119652 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:12:56.168161 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:12:56.214987 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:12:56.224773 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:12:56.259824 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:12:56.290762 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:12:56.492789 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:56.492884 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:12:56.539187 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:12:56.555367 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:56.588955 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:12:56.604652 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:56.610342 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:12:56.660606 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:12:56.673909 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:12:56.689356 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:12:56.730580 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:12:56.754662 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:12:56.755653 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:12:56.795413 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:12:56.796679 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:12:56.976250 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:12:56.984975 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:57.041694 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:57.081885 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:12:57.150437 4931 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 14:12:57.185700 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:57.219599 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240501 14:12:57.328423 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:12:57.384265 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:12:57.427937 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:12:57.433806 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
I20240501 14:12:57.456064 13035 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:12:57.456241 13035 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:12:57.457070 13035 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:12:57.457342 5255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" is_pre_election: true
I20240501 14:12:57.457590 5255 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 2.
W20240501 14:12:57.458634 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:12:57.458832 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:12:57.458915 5321 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:12:57.459147 13035 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:12:57.550732 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:12:57.586498 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:12:57.606515 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:57.612147 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:57.613770 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:12:57.627533 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:12:57.686606 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:12:57.723778 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:12:57.759076 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:12:57.827067 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:12:57.838460 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:12:57.887573 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:12:57.888711 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:12:57.956115 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:57.971773 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:12:58.001750 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 142 similar messages]
W20240501 14:12:58.005748 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:58.020869 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:12:58.079413 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:58.093551 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:12:58.261862 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 129 similar messages]
W20240501 14:12:58.318647 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:12:58.431152 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:12:58.464727 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:58.495069 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:58.496459 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:12:58.512461 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:12:58.545833 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:12:58.608479 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 142 similar messages]
W20240501 14:12:58.635718 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:12:58.658442 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:12:58.674337 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:58.730067 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:12:58.840000 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:12:58.870620 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:12:58.905437 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:12:58.906495 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:58.940610 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:12:58.993155 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:12:58.997907 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:12:59.012658 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:59.073228 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:12:59.087358 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:12:59.188211 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:12:59.210202 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:59.225036 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:12:59.232678 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:12:59.246798 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:12:59.265733 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:12:59.328303 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:12:59.401955 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:12:59.467391 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:59.492959 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:12:59.512123 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:59.552573 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:12:59.603037 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:59.716795 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:12:59.945390 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:12:59.972992 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240501 14:12:59.992059 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:12:59.995429 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:13:00.034958 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:00.039187 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:13:00.074999 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:13:00.081931 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:13:00.118393 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:13:00.131857 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:13:00.137845 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:13:00.149293 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:13:00.184278 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:13:00.221379 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:13:00.228070 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:13:00.342583 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:00.413632 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:13:00.474329 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:13:00.478811 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:00.565611 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:13:00.579509 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:00.630380 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:13:00.724710 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:13:00.730693 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:13:00.783303 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:00.850697 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:13:00.861418 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:13:00.893836 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:13:00.902431 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:13:00.923949 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:01.064172 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:01.105113 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:13:01.146159 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:13:01.252782 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:01.307660 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:01.325300 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:13:01.374773 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:13:01.394742 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:13:01.429492 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:13:01.433754 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:13:01.464712 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:13:01.520376 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:13:01.526307 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:13:01.555583 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
I20240501 14:13:01.608552 13210 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:13:01.608747 13210 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:13:01.609840 13210 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers be56773105e744919601127a94e98adf (127.29.124.132:35459), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:13:01.610929 5384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf" is_pre_election: true
I20240501 14:13:01.611117 5384 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 2.
W20240501 14:13:01.614466 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:13:01.614717 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:13:01.614809 5191 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:13:01.615051 13210 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:13:01.621136 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:13:01.629094 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:01.631704 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:13:01.713764 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:13:01.812417 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:01.817705 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:13:01.818101 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:13:01.819372 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:13:01.832788 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:01.839721 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:13:01.896237 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:13:01.959726 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:13:02.098731 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:02.156037 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:13:02.216070 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 14:13:02.249831 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:02.278649 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:02.360365 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:13:02.365214 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:13:02.460220 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:13:02.472164 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:13:02.519168 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:13:02.530136 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:13:02.641492 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:02.695739 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:13:02.707918 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:13:02.744913 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:02.785523 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:13:02.796880 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:13:02.822882 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:02.828315 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:13:02.836083 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240501 14:13:02.857848 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:13:02.923607 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:13:02.924521 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:13:02.970227 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:13:03.049753 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:13:03.064365 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 142 similar messages]
W20240501 14:13:03.107057 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:13:03.116554 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:13:03.181892 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:03.252651 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:03.277145 5192 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 135 similar messages]
W20240501 14:13:03.287286 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:13:03.396080 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:13:03.401155 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:13:03.412245 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:13:03.436033 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:03.619576 5062 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240501 14:13:03.666028 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:13:03.744455 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:03.816546 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:03.841320 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:13:03.841416 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:13:03.853338 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:03.860126 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:13:03.866742 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:13:03.891198 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:13:03.903615 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:13:03.998229 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:13:04.008381 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:13:04.065924 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:13:04.131266 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:13:04.169027 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:04.194254 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:13:04.247888 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:13:04.312214 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:13:04.325471 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:04.366173 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:13:04.401060 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:13:04.418581 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:04.423734 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:13:04.456292 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:13:04.475788 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:13:04.554598 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:13:04.646514 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:13:04.762226 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:04.780117 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:04.782910 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:13:04.884728 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:04.887149 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:13:05.035847 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:13:05.145040 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:13:05.171123 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:13:05.207767 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:13:05.217576 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:13:05.221196 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:05.248986 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:13:05.267263 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:13:05.326818 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:05.349759 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240501 14:13:05.361346 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:05.384999 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:13:05.397827 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:13:05.408881 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:13:05.440308 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:13:05.446681 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:13:05.481361 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:13:05.691345 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:13:05.732580 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:05.788779 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:13:05.841152 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:13:05.842327 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:13:05.846104 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:05.848280 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:13:05.909361 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:13:05.957957 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:06.058303 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:13:06.112701 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:13:06.233404 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:06.237671 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:06.242286 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:13:06.309993 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:13:06.330475 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:13:06.413061 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:13:06.450990 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:13:06.480741 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:13:06.503679 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:13:06.538153 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:06.556138 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:13:06.649014 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:13:06.681447 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:06.708863 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:13:06.766287 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:13:06.781320 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:06.796466 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:13:06.813658 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:13:06.814814 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:13:06.881495 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:13:06.956008 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:06.958053 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:13:06.996796 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:13:07.014789 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:13:07.091567 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:13:07.204210 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:07.257019 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:07.257200 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:13:07.339403 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:13:07.344448 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:13:07.451479 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:07.481029 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 14:13:07.523716 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:13:07.591149 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:13:07.716120 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:07.732872 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:13:07.742784 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:13:07.749867 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:13:07.763489 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:07.794139 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:13:07.819613 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240501 14:13:07.844292 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:13:07.874853 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:13:07.876695 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:13:07.917521 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:08.073789 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:13:08.092288 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:13:08.136257 5322 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240501 14:13:08.167219 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:13:08.200695 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:13:08.238093 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:13:08.238375 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:08.287797 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:13:08.290992 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:13:08.292872 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 126 similar messages]
W20240501 14:13:08.293792 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:08.324187 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:13:08.381824 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:13:08.499251 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:08.553699 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:13:08.589205 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:13:08.645844 5062 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240501 14:13:08.688083 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:08.797456 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:13:08.830904 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:13:08.883934 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:08.912933 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:13:08.917486 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:13:08.996809 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:13:09.114023 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:09.139364 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:09.161883 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:13:09.197083 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:13:09.263571 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:09.307927 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:13:09.328375 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:13:09.366086 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:13:09.382628 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:13:09.416267 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:13:09.467196 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:13:09.478746 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:13:09.504581 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:13:09.552961 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:13:09.580327 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:13:09.607993 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:13:09.623275 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:13:09.625783 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:09.679000 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:13:09.681668 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:09.727612 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:13:09.769722 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:09.881464 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:13:09.985805 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:13:10.030894 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:10.102048 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:10.160491 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:13:10.162374 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:13:10.245710 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:13:10.328084 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:10.328785 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:13:10.360083 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:13:10.407644 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:13:10.429836 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:13:10.477449 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240501 14:13:10.527541 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:13:10.583352 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:13:10.602984 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:13:10.625591 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:10.638162 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:13:10.680814 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:13:10.698757 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:10.710484 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:13:10.756721 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:10.878517 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:13:10.979224 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:13:10.997725 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:13:11.014253 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:13:11.106894 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:13:11.165169 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:13:11.248322 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:11.248703 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:13:11.251014 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:13:11.265873 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:13:11.300056 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:13:11.309191 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:11.312197 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:11.483172 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:13:11.565184 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:13:11.646392 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:13:11.705226 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:11.750205 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:11.773247 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:11.852970 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:13:11.859751 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:13:11.920001 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:13:11.949177 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:13:11.968677 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:13:11.986208 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:13:12.007071 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:13:12.028648 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:13:12.030769 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:13:12.043936 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:13:12.139082 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:13:12.168736 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:13:12.201375 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:13:12.226966 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:12.239681 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:12.240617 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:13:12.252462 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:12.309003 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:13:12.440428 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:13:12.468586 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:13:12.631655 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:13:12.634250 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:13:12.654421 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240501 14:13:12.658288 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:13:12.702577 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:12.706166 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240501 14:13:12.715540 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:13:12.722564 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:13:12.748822 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:12.813596 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:12.914209 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:13:12.983953 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:13:12.990986 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:13:13.125887 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:13:13.151147 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240501 14:13:13.192755 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:13:13.195277 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:13:13.199896 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:13:13.232012 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:13.269377 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:13:13.288002 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:13.300697 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:13:13.383009 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240501 14:13:13.383896 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:13.489563 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:13:13.560493 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:13:13.565593 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:13:13.603188 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:13:13.693666 5062 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 145 similar messages]
W20240501 14:13:13.703248 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:13:13.708726 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:13:13.712636 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:13.716853 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:13:13.730046 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:13:13.842119 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:13.945868 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:13.961896 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:13:14.046401 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:13:14.204947 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:13:14.256037 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:14.268033 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:13:14.313549 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:14.365116 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:13:14.369132 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:13:14.457583 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:13:14.487969 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:13:14.519501 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:13:14.528892 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:14.544369 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:13:14.572729 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:13:14.575425 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:13:14.641827 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:13:14.661129 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:14.680588 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:13:14.789721 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:13:14.790169 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:13:14.823277 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:13:14.838064 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:14.887092 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:13:14.897214 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:13:14.921684 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:13:14.942551 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:13:14.947892 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:13:14.969874 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:15.042454 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:13:15.096277 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:13:15.110658 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:15.148766 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:13:15.172833 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:13:15.305606 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240501 14:13:15.396093 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:13:15.428478 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:15.440968 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:15.475214 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:13:15.479030 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:13:15.502874 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:15.513504 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:13:15.605734 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:13:15.780809 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:13:15.781216 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:13:15.794231 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:13:15.862690 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:13:15.911698 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:15.992554 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:13:16.006353 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:16.039013 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:16.060660 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:13:16.115828 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:13:16.164316 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:13:16.185667 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:13:16.191978 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:13:16.376900 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:13:16.435307 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:16.461711 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:13:16.484774 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:13:16.532536 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:16.538538 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:16.591288 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:13:16.695549 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:13:16.786862 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:13:16.821930 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:13:16.865309 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:16.941520 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:13:16.951431 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:16.965471 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:13:16.979166 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:17.005000 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:13:17.025904 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:13:17.055541 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:13:17.068025 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:13:17.158249 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:13:17.176038 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:13:17.204574 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:13:17.211174 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:13:17.268119 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:13:17.291565 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:13:17.293565 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:17.324203 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:13:17.327361 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:17.481639 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:13:17.533026 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:13:17.535331 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:17.564487 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:13:17.639598 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:13:17.656333 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 14:13:17.659204 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:13:17.671969 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:13:17.698244 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240501 14:13:17.704598 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:17.720755 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:17.738732 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:13:17.741466 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:13:17.783921 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:13:17.920755 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:17.973683 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:13:18.039183 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:13:18.077903 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:13:18.140872 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:13:18.195367 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:13:18.197947 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:13:18.217479 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:18.247431 5322 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240501 14:13:18.295177 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:18.361442 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:13:18.412385 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240501 14:13:18.438540 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:18.441776 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:13:18.474874 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:13:18.549527 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:13:18.615056 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:13:18.694082 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240501 14:13:18.730747 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:18.752056 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:13:18.786428 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:13:18.861071 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:13:18.885452 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:18.915851 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:18.928397 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:13:19.056708 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:13:19.066998 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
I20240501 14:13:19.085269 14273 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:13:19.085410 14273 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:13:19.086119 14273 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:13:19.086380 5255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" is_pre_election: true
I20240501 14:13:19.086565 5255 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 2.
W20240501 14:13:19.087370 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:13:19.087540 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:13:19.087622 5321 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:13:19.087838 14273 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:13:19.133320 4838 catalog_manager.cc:5014] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet de6c7753c8564107b858876fcfa5924d with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER failed with leader 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }.
Pending config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } }
W20240501 14:13:19.146179 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:19.301172 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:19.303637 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:13:19.320768 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:13:19.347186 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:13:19.440524 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:19.450702 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:13:19.575984 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:13:19.584312 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:13:19.643231 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:13:19.646175 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:13:19.706226 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:13:19.736157 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:19.759192 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:13:19.768383 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:13:19.776511 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:13:19.866914 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:13:19.903216 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:13:19.908118 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:19.923717 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:13:19.928189 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:13:19.952513 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:19.955442 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:13:19.993229 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:13:20.047371 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:13:20.191110 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:13:20.229489 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:13:20.249498 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:13:20.254607 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:13:20.260155 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:13:20.262490 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:20.284978 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:20.294466 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240501 14:13:20.341764 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:13:20.435213 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:13:20.449844 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:20.590226 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:13:20.630661 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:13:20.666589 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:13:20.781569 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:13:20.847868 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:20.859835 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:20.921425 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:13:20.926188 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:13:20.933216 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:13:20.990700 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:21.017206 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:13:21.190225 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:13:21.298202 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:21.340575 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:13:21.345727 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:13:21.346897 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:13:21.406883 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:13:21.417241 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:21.470995 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:21.518952 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:13:21.590036 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:13:21.773797 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:13:21.820832 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:13:21.827539 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:13:21.837924 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:21.865154 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:13:21.983738 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:13:21.988847 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:22.068470 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:22.080365 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:13:22.205724 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:13:22.213140 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:13:22.234180 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:13:22.254506 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:13:22.280715 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:13:22.301571 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:13:22.329361 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:13:22.376720 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:13:22.390962 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:13:22.393875 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:13:22.417762 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:22.418200 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:13:22.425966 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:22.473536 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:13:22.483939 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:22.513311 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:13:22.513331 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:13:22.663115 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:13:22.715323 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:13:22.718408 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240501 14:13:22.724959 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 14:13:22.726023 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:13:22.730285 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:13:22.794831 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:13:22.811887 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:13:22.844403 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:13:22.859540 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:22.974016 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:22.993582 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:23.025082 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:13:23.178301 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:13:23.184046 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:13:23.262651 5322 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 137 similar messages]
W20240501 14:13:23.263760 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:13:23.299747 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:23.340454 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:13:23.408764 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:23.414458 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:13:23.447939 5192 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 128 similar messages]
W20240501 14:13:23.513792 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:13:23.522236 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:23.706370 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 137 similar messages]
W20240501 14:13:23.709867 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:13:23.754894 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:23.767578 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:13:23.861027 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:13:23.983837 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:24.010895 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:13:24.022228 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:13:24.049532 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:24.053447 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:13:24.072088 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:13:24.168273 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:24.219035 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:13:24.358639 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:13:24.384979 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:13:24.400161 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:13:24.404971 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:13:24.408828 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:24.426523 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:13:24.454445 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:13:24.460994 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:24.515905 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:13:24.521932 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:13:24.618813 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:13:24.638303 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:13:24.708052 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:13:24.743763 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:13:24.745532 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:13:24.757920 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:24.775519 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:13:24.779723 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:13:24.789167 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:13:24.857784 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:13:24.864475 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:13:24.884645 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:13:24.901003 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:24.937939 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:25.046844 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:13:25.126693 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:13:25.172722 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:25.211133 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:13:25.240442 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:13:25.242879 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:13:25.250703 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:13:25.306885 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:25.431834 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:13:25.438967 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:13:25.458222 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240501 14:13:25.552220 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:13:25.561997 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:25.575260 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
I20240501 14:13:25.594061 14296 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:13:25.594259 14296 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:13:25.595085 14296 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers be56773105e744919601127a94e98adf (127.29.124.132:35459), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:13:25.595448 5384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf" is_pre_election: true
I20240501 14:13:25.595670 5384 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 2.
W20240501 14:13:25.596756 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:13:25.596944 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:13:25.597033 5191 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:13:25.597240 14589 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:13:25.599360 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:13:25.743378 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:13:25.894387 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:25.902648 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:13:25.934262 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:13:26.009173 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:26.013573 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:26.030725 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:13:26.066268 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:13:26.305205 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:13:26.441780 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:13:26.485399 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:13:26.506354 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:26.508133 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:26.542455 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:13:26.565878 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:13:26.614511 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:13:26.627707 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:26.704939 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:13:26.770470 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:13:26.800537 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:13:26.876204 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:13:26.918982 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:13:26.927191 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:13:26.932464 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:13:26.964939 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:13:26.990149 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:27.010550 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:27.041064 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:13:27.072582 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:13:27.079829 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:13:27.113663 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:13:27.116514 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:13:27.127630 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:27.196427 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:13:27.223692 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:13:27.285372 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:13:27.289712 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:13:27.298745 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:13:27.315629 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:13:27.345667 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:13:27.364167 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:13:27.410869 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:13:27.441911 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:13:27.442124 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:27.522056 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:27.536377 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:27.679364 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:13:27.694972 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:13:27.712476 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:13:27.742687 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240501 14:13:27.743500 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:13:27.840075 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:13:27.930544 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:27.937062 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:13:28.002552 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:13:28.046682 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:28.088021 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:28.168689 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:13:28.250689 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240501 14:13:28.268116 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:13:28.316576 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:28.330245 5322 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240501 14:13:28.347509 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:13:28.395987 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:13:28.512346 5192 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 128 similar messages]
W20240501 14:13:28.514596 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:28.517724 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:13:28.537829 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:13:28.677809 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:28.719794 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240501 14:13:28.785697 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:13:28.817181 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:13:28.858873 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:28.942113 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:13:28.954038 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:28.998054 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:13:29.143606 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:29.158376 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:13:29.186345 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:13:29.205354 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:13:29.215556 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:13:29.259588 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:13:29.265190 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:29.297016 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:13:29.313042 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:13:29.362697 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:29.363910 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:13:29.368999 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:13:29.546082 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:13:29.546149 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:13:29.565685 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:13:29.592433 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:13:29.600220 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:13:29.672392 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:13:29.678107 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:13:29.713372 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:13:29.736295 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:29.758443 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:13:29.795506 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:29.824900 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:13:29.847357 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:13:29.941514 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:29.945230 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:13:29.991884 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:13:30.020823 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:13:30.042634 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:13:30.076362 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:13:30.094451 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:13:30.153146 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:13:30.196961 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:13:30.313766 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:30.316957 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:30.390744 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:13:30.400076 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:13:30.415043 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:13:30.426481 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:13:30.437851 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:30.445582 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:13:30.527696 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240501 14:13:30.862493 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:30.896349 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:13:30.933202 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:30.967490 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:13:30.977507 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:13:30.980345 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:31.006675 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:13:31.187299 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:13:31.283397 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:13:31.344154 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:13:31.374037 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:31.393517 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:13:31.425752 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:13:31.515517 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:31.516541 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:13:31.538177 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:31.637115 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:13:31.697213 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:13:31.764750 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:13:31.792922 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:13:31.799100 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:13:31.808405 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:13:31.899299 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:13:31.913116 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:13:31.934506 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:31.953904 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:13:31.969101 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:31.983426 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:13:32.019045 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:13:32.019675 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:32.045648 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:13:32.101414 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:13:32.111399 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:13:32.120388 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:13:32.122572 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:13:32.136706 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:13:32.425611 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:13:32.449844 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:13:32.450313 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:13:32.471330 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:13:32.515751 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:32.552987 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:32.553997 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:32.561842 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:13:32.565454 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:13:32.582994 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:13:32.605067 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:13:32.667809 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:13:32.750486 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:13:32.784004 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240501 14:13:32.805786 4931 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 14:13:32.872922 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:13:32.934787 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:13:32.936496 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:13:32.999655 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:33.020471 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:33.034448 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:13:33.054291 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:13:33.121999 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:33.220285 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:13:33.319468 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:13:33.336697 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240501 14:13:33.395421 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:13:33.443055 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:13:33.489217 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:33.533313 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 128 similar messages]
W20240501 14:13:33.538936 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
I20240501 14:13:33.602576 14751 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:13:33.602783 14751 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:13:33.603570 14751 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:13:33.604250 5254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" is_pre_election: true
I20240501 14:13:33.604511 5254 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 2.
W20240501 14:13:33.606639 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:13:33.606798 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:13:33.606875 5321 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:13:33.607095 14751 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:13:33.615893 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:13:33.623432 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:33.769469 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 142 similar messages]
W20240501 14:13:33.881440 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:13:33.891278 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:13:33.903888 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:13:33.931640 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:13:34.046473 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:34.047364 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:34.083535 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:13:34.097937 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:13:34.204113 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:34.231251 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:13:34.253652 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:13:34.272047 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:13:34.366286 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:13:34.373287 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:13:34.373700 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:13:34.382803 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:13:34.434844 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:13:34.484686 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:13:34.510593 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:34.571138 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:13:34.577704 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:13:34.594506 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:13:34.613842 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:34.617640 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:13:34.645287 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:13:34.761135 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:13:34.774667 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:34.879843 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:13:34.943898 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:13:35.047145 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:13:35.066771 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:35.074055 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:35.101660 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:13:35.105032 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:13:35.136945 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:13:35.221218 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:13:35.221681 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:13:35.264819 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240501 14:13:35.300338 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:13:35.302146 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:13:35.319780 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:13:35.370630 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:13:35.399155 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:35.508227 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:13:35.528771 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:35.544714 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:13:35.548947 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:13:35.624679 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:13:35.679127 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:35.690210 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:13:35.805236 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:13:35.824895 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:35.921221 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:35.926213 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:13:36.052187 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:13:36.135964 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:13:36.260366 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:36.278970 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:13:36.353179 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:13:36.394507 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:36.410449 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:13:36.416290 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:13:36.457648 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:36.550789 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:13:36.553488 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:13:36.670542 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:13:36.688825 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:13:36.689116 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:13:36.694346 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:13:36.704385 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:13:36.759676 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:13:36.795890 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:36.892755 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:36.909168 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:36.941035 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:13:36.941097 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:13:37.071476 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:13:37.125159 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:13:37.158902 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:13:37.224694 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:13:37.261324 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:13:37.283151 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:37.287406 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:37.297725 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:37.349381 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:13:37.356478 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:13:37.480489 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:13:37.534929 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:13:37.583937 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:13:37.594044 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:13:37.664268 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:13:37.693557 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:37.704464 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240501 14:13:37.724191 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:13:37.726001 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:37.818243 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 14:13:37.833715 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:37.875972 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:13:37.882020 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:13:37.896385 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:13:37.914884 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:13:37.951879 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:13:38.001447 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:13:38.041559 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:13:38.080159 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:13:38.123663 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:13:38.138739 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:38.142096 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:13:38.204118 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:13:38.301431 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:38.304363 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:13:38.306697 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:13:38.327757 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:38.393342 5322 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 142 similar messages]
W20240501 14:13:38.467291 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:13:38.552035 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 128 similar messages]
W20240501 14:13:38.616298 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:13:38.624948 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:13:38.653332 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:38.726197 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:13:38.731177 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:13:38.775604 5062 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240501 14:13:38.793569 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:38.812650 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:13:38.819903 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:38.870967 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:13:38.919907 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:13:38.932794 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:13:39.010778 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:13:39.028906 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:13:39.040957 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:13:39.200946 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:39.246721 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:39.264649 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:13:39.362582 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:39.387027 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:13:39.442369 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:13:39.444399 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:13:39.538071 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:13:39.543354 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:13:39.656884 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:39.687323 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:39.689471 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:13:39.744923 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:13:39.846882 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:13:39.852577 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:13:39.941201 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:39.942138 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:13:39.985366 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:13:39.995612 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:13:40.063052 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:13:40.075680 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:13:40.118294 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:13:40.163769 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:40.195930 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:40.199919 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:13:40.218401 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:13:40.404253 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240501 14:13:40.408248 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:13:40.430152 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:40.460270 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:13:40.460680 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:13:40.551992 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:13:40.580483 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:13:40.585031 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:13:40.598245 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:40.598284 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:13:40.648324 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:13:40.747346 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:40.762697 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:13:40.793551 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:13:40.825661 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:40.896566 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:13:40.907528 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:13:40.914134 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:13:40.954272 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:13:40.995184 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:41.087049 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:13:41.090880 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:13:41.156152 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:13:41.159297 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:13:41.228650 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:41.371413 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:41.396521 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:41.482539 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:13:41.499748 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:13:41.502842 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:13:41.539129 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:13:41.651178 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:13:41.679149 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:13:41.705183 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:41.828420 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:41.885080 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:13:41.903800 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:13:41.907295 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:13:41.920476 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:41.931754 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:13:41.942834 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:13:42.074867 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:13:42.105427 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:13:42.153198 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:13:42.156056 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:13:42.211668 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:13:42.217993 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:42.246850 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:42.277374 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:13:42.308650 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:13:42.417493 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:42.537488 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:13:42.684275 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:13:42.709786 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:13:42.739221 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:13:42.776161 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:13:42.787598 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:13:42.794709 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:42.839545 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:42.847796 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 14:13:42.850744 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:42.915416 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:13:43.009703 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:13:43.066540 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:13:43.069108 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:13:43.074708 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:13:43.100900 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240501 14:13:43.114378 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:13:43.114539 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:13:43.204787 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
I20240501 14:13:43.225970 15477 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:13:43.226138 15477 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:13:43.226863 15477 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers be56773105e744919601127a94e98adf (127.29.124.132:35459), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:13:43.227326 5386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf" is_pre_election: true
I20240501 14:13:43.227559 5386 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 2.
W20240501 14:13:43.228300 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:13:43.228494 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:13:43.228583 5191 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:13:43.228813 15477 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:13:43.272490 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:43.294417 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:13:43.302513 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:43.305537 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:13:43.321681 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:13:43.375707 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:43.397114 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:13:43.458084 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:13:43.467437 5322 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240501 14:13:43.505313 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:13:43.519654 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:13:43.589222 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240501 14:13:43.684631 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:13:43.723263 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:13:43.782713 5062 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240501 14:13:43.819142 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:13:43.860769 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:43.864651 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:43.904263 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:13:43.953789 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:43.957162 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:13:43.964314 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:13:44.013765 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:13:44.205137 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:13:44.213889 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:13:44.340054 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:44.412000 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:13:44.431304 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:44.462477 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:44.471096 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:13:44.485963 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:13:44.486366 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:13:44.502132 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:13:44.556725 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:13:44.595140 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:13:44.610913 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:13:44.625241 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:13:44.758850 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:13:44.869866 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:13:44.875967 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:13:44.882228 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:44.889423 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:44.911195 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:44.971586 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:13:45.001536 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:13:45.198220 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:13:45.223419 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:13:45.274349 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:13:45.349907 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:45.384346 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:45.425488 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:13:45.439430 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:45.512253 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:13:45.600538 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:13:45.673249 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:13:45.695596 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:13:45.697773 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:13:45.743738 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:13:45.758194 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:13:45.760883 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240501 14:13:45.824497 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:13:45.829391 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:13:45.832784 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:13:45.843665 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:45.844493 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:13:45.863166 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:45.891678 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:13:45.923588 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:13:45.973213 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:13:45.991164 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:46.105247 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:13:46.141654 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:13:46.212464 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:13:46.242051 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:46.250229 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:13:46.323382 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:13:46.367727 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:46.422086 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:13:46.432828 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:46.562127 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:13:46.643908 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:13:46.676010 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:46.770355 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:46.771868 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:13:46.773317 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:13:46.814982 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:13:46.845237 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:46.848429 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:13:46.916065 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:13:46.995252 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:13:46.998692 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:13:47.048857 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:13:47.077314 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:13:47.112573 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:13:47.122651 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:47.177439 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:13:47.345714 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:47.361320 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:13:47.374105 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:13:47.395066 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:13:47.403079 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:47.509840 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:13:47.575685 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:13:47.589912 4840 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 13 similar messages]
W20240501 14:13:47.590916 4840 catalog_manager.cc:4573] TS 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9d32e0707d244f2c9bbcb57f62b09f31: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:13:47.620962 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:47.753696 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:13:47.765607 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:13:47.825627 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:13:47.842494 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:13:47.884620 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 14:13:47.887765 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:47.935132 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:48.018636 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:13:48.023284 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:13:48.042454 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:13:48.078384 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:13:48.112231 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:13:48.134696 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:48.149783 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:13:48.185017 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:13:48.211005 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240501 14:13:48.282747 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:13:48.283581 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:13:48.356706 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:48.370965 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:48.383013 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:13:48.400166 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:13:48.417934 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:13:48.451792 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:13:48.478353 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240501 14:13:48.479128 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:13:48.480304 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:13:48.555696 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:13:48.566078 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:13:48.570546 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:48.626956 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 132 similar messages]
W20240501 14:13:48.759006 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:48.779299 4840 catalog_manager.cc:4573] TS 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 61a274b24f53458f88c2cde4f8dbe4f4: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:13:48.806660 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:48.811051 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240501 14:13:48.881182 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:13:48.905561 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:13:48.947779 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:13:49.050756 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:49.115953 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:13:49.122716 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:13:49.169487 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:13:49.198307 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:49.273695 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:13:49.276894 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:13:49.323669 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:13:49.330922 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:13:49.394748 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:49.425460 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:13:49.525877 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:13:49.529891 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:49.583971 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:13:49.590801 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:13:49.612372 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:13:49.739653 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:49.786486 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:13:49.801121 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:13:49.820881 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:49.851327 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:13:49.930255 4840 catalog_manager.cc:4573] TS 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet fee16905387d428aa02cd2070c86c790: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:13:49.972084 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:49.993820 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:13:50.006803 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:13:50.022318 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:13:50.022891 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:13:50.133366 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:13:50.184403 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:50.211524 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:50.268621 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:13:50.365034 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:13:50.494527 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:13:50.559486 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:50.561226 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:13:50.578990 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:50.660607 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:13:50.675395 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:13:50.678037 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:13:50.702046 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:13:50.753049 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:13:50.767385 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:50.797135 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:13:50.819435 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:13:50.826210 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:13:50.860373 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:13:50.869851 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:13:50.885797 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240501 14:13:50.905623 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:13:50.916810 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:13:51.052659 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:13:51.058490 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:51.102795 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:13:51.137468 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:13:51.158926 4840 catalog_manager.cc:4573] TS 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c8a961e711114f369ba06840c062b982: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:13:51.188509 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:51.336941 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:51.413795 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:13:51.432400 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:13:51.440776 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:13:51.521566 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:51.574636 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:51.740633 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:13:51.766161 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:13:51.775691 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:13:51.817000 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:13:51.862255 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:13:51.941217 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:51.943533 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:13:51.961427 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:13:51.976333 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:13:52.026835 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:52.039230 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:13:52.049074 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:52.070927 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:13:52.178870 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:13:52.203337 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:13:52.218514 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:13:52.229214 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:13:52.236380 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:13:52.350912 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:13:52.403208 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:13:52.490576 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:52.529639 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:52.529708 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:13:52.548748 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:52.556008 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:13:52.607316 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:13:52.616259 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:13:52.808209 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:13:52.915184 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:13:52.922240 4931 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 14:13:52.937140 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:52.949641 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:52.969918 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:13:52.986733 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:53.001349 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:13:53.022732 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:13:53.175101 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:13:53.200724 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:13:53.246819 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:13:53.291054 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:13:53.302693 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:13:53.384435 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:13:53.396508 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:53.406363 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:13:53.413131 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:13:53.443706 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:13:53.445348 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:13:53.464876 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240501 14:13:53.483959 5322 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 143 similar messages]
W20240501 14:13:53.484673 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:53.491629 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:53.596885 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:13:53.624603 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:13:53.672410 5192 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 126 similar messages]
W20240501 14:13:53.804312 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:53.807421 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:13:53.836539 5062 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 142 similar messages]
W20240501 14:13:53.878131 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:13:53.908424 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:13:53.914793 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:53.957899 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:54.130563 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:13:54.171715 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:13:54.176535 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:13:54.177294 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:13:54.211268 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:13:54.230733 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:54.240939 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:13:54.301376 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:13:54.310528 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:13:54.364588 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:54.498485 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:54.519788 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:13:54.539006 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:13:54.589443 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:13:54.614943 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:13:54.642266 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:13:54.708221 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:13:54.718116 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:13:54.814225 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:54.815460 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:13:54.824373 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:13:54.848500 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:13:54.887339 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:13:54.891963 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:55.017524 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:13:55.107641 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:55.157512 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:13:55.158444 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:13:55.207808 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:13:55.224969 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:13:55.301118 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:55.427683 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:55.436784 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:13:55.489768 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:55.543566 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:13:55.551528 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:13:55.577445 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:13:55.597764 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:13:55.758028 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:13:55.759567 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:55.788430 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:13:55.811462 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:13:55.842394 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:13:55.877518 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:13:55.884161 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:13:55.924330 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:13:55.972241 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:55.987095 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:56.020534 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:13:56.048456 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:13:56.055814 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:13:56.220031 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240501 14:13:56.229528 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:13:56.330897 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:56.364869 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:13:56.366279 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:13:56.498044 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:56.536187 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:56.579795 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:13:56.588316 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:13:56.674950 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:13:56.679893 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:13:56.789436 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:13:56.814893 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:13:56.926002 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:56.966398 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:13:56.978932 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:13:57.007650 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:13:57.034068 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:57.057061 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:13:57.064021 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:57.084403 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:13:57.124406 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:13:57.147697 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:13:57.170928 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:13:57.186013 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:13:57.325460 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:13:57.336319 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
I20240501 14:13:57.381531 16262 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:13:57.381690 16262 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:13:57.382407 16262 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers be56773105e744919601127a94e98adf (127.29.124.132:35459), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:13:57.382826 5386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf" is_pre_election: true
I20240501 14:13:57.383035 5386 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 2.
W20240501 14:13:57.384387 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:13:57.384531 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:13:57.384670 5191 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:13:57.384886 16262 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:13:57.413210 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:13:57.415383 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:13:57.426972 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:57.450911 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:57.482087 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:13:57.496714 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:13:57.537339 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:13:57.565491 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:57.682812 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:13:57.835594 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:13:57.835702 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:57.844614 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:13:57.908874 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:13:57.931849 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:13:57.947154 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:13:57.977174 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:58.002429 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:58.032436 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 14:13:58.045445 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:13:58.144623 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:13:58.181118 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:13:58.189898 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:13:58.225399 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:13:58.353508 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:13:58.355006 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:58.465677 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:13:58.500965 5322 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 138 similar messages]
W20240501 14:13:58.501855 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:58.526068 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:13:58.554857 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:13:58.566754 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:58.573650 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:13:58.728689 5192 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 131 similar messages]
W20240501 14:13:58.729688 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:13:58.772569 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:58.788368 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:13:58.935353 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240501 14:13:58.936313 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:13:58.990134 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240501 14:13:59.027140 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:13:59.041903 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:13:59.058883 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:59.075345 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:59.085448 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:13:59.209826 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:13:59.244028 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:59.317680 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:13:59.344341 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:13:59.364861 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:13:59.403035 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:13:59.482985 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:13:59.489362 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:59.509728 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:13:59.514196 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:59.576551 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:13:59.595425 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:13:59.631600 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:13:59.647436 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:13:59.654780 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:13:59.745059 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:13:59.747702 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:13:59.878422 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:13:59.891849 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:13:59.940189 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:13:59.968590 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:14:00.064004 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:14:00.069480 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:14:00.080919 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:14:00.106864 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:00.130520 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:14:00.144254 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:14:00.146714 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:14:00.224130 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:00.327093 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:14:00.350939 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:00.375854 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:14:00.475399 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:14:00.496510 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:14:00.530145 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:14:00.551532 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:00.574447 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:14:00.599448 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:14:00.722757 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:00.765247 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:14:00.855507 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:14:00.880800 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:00.982281 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:01.027436 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:14:01.098338 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:14:01.139623 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:01.167896 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:14:01.189072 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240501 14:14:01.196389 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:14:01.204772 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:14:01.283310 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:14:01.320168 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:14:01.417620 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:14:01.421298 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:01.522341 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:14:01.535627 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:14:01.576226 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:01.601943 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:01.700637 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:14:01.771595 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:14:01.788988 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:14:01.813373 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:14:01.881536 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:01.942140 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:14:01.945564 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:14:01.973022 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:14:02.008986 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:14:02.009862 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:02.051802 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:14:02.083801 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:02.100407 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:14:02.156030 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:14:02.160176 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:14:02.218189 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:14:02.288441 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:14:02.308531 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:14:02.313983 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:14:02.389521 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:02.428788 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:14:02.501416 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:02.509747 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:14:02.512310 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:14:02.577296 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:14:02.577689 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:02.587083 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:14:02.755548 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:14:02.765316 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:14:02.809578 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:02.903421 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:14:02.930219 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:14:02.951900 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:14:03.025768 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:14:03.038681 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:14:03.079130 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:03.120311 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 14:14:03.121323 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:14:03.148001 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:14:03.180908 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:03.230002 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:03.240410 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:14:03.317713 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:14:03.400110 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:14:03.484066 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:03.536484 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240501 14:14:03.661881 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:14:03.722465 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:14:03.734707 5192 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240501 14:14:03.768898 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:14:03.804319 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:03.816782 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:03.822575 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:14:03.822880 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:14:03.840507 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240501 14:14:03.867724 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:14:03.905566 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:14:03.970002 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240501 14:14:03.980351 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:14:04.072731 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:14:04.085628 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:04.130743 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:14:04.138525 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:14:04.205361 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:04.233629 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:04.342490 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:14:04.425204 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:14:04.496693 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:14:04.540799 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:14:04.559962 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:14:04.572069 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:14:04.598997 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:14:04.612991 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:14:04.613171 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:04.620276 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:14:04.652840 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:14:04.692898 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:04.696580 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:04.725374 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:14:04.774353 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:14:04.775936 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:14:04.880615 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:14:04.888412 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:14:04.999573 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:14:05.067863 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:14:05.074916 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:05.125509 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:05.138329 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:14:05.202536 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:14:05.233968 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:05.272414 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:14:05.342113 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:14:05.350880 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:14:05.353904 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:14:05.365540 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:14:05.528062 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:05.550264 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:14:05.571105 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:14:05.608042 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:14:05.658201 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:05.709925 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:14:05.714406 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:14:05.792151 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:14:05.819384 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:14:05.853667 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:06.069608 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:06.122617 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:06.131352 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:14:06.145893 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:14:06.146160 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240501 14:14:06.174257 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:14:06.228317 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:14:06.361811 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:06.381740 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:14:06.387506 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:14:06.394762 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:14:06.399785 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:14:06.498111 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:06.562455 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:14:06.565795 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:14:06.569972 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:06.633080 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:14:06.735857 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:14:06.866716 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:14:06.878077 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:06.878851 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:06.910730 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:14:07.013794 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:14:07.033628 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:14:07.044777 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:14:07.071369 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:14:07.135043 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:14:07.155092 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:14:07.159724 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:07.162650 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:14:07.268246 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:14:07.295766 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:14:07.425756 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:14:07.442924 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:07.468271 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:07.500481 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:14:07.516348 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:14:07.566008 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:14:07.590173 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:14:07.619694 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:14:07.680930 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:14:07.687598 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:14:07.725414 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:07.766181 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:14:07.821442 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:14:07.909632 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:14:07.938284 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:14:07.987368 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:08.001425 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:08.057323 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:14:08.103001 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:14:08.183400 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:14:08.210036 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 14:14:08.241647 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:14:08.251173 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:14:08.263662 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:08.360096 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:14:08.379033 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:14:08.472815 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:14:08.525866 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240501 14:14:08.526865 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:14:08.537735 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240501 14:14:08.562002 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:14:08.568212 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:08.583184 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:08.744200 5192 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 128 similar messages]
W20240501 14:14:08.745144 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:08.756098 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:14:08.827978 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:14:08.838196 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:14:08.946395 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:14:08.981222 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:14:08.983599 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 142 similar messages]
W20240501 14:14:08.984261 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:09.071390 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:09.111812 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:14:09.155798 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:14:09.214592 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:14:09.215001 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:09.328608 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:14:09.400482 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:09.410928 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:14:09.427323 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:14:09.430495 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:14:09.452209 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:14:09.489957 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:14:09.517325 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:09.550889 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:14:09.633069 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:14:09.678210 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:14:09.816560 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:09.835171 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:14:09.854478 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:14:09.890061 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:14:09.899837 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:14:09.916047 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:09.923168 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:14:09.923869 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:14:09.976919 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:14:10.031069 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:14:10.044016 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:14:10.065779 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:14:10.083369 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:14:10.116873 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:10.221457 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:10.281090 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:14:10.467527 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:14:10.473093 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:14:10.481628 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:10.505342 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:14:10.532516 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:14:10.566485 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:14:10.580068 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:14:10.580077 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:10.583223 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:14:10.635877 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:10.635877 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:14:10.894450 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:14:10.934469 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:14:10.960844 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:14:10.992020 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:10.995743 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:14:10.997246 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:11.069861 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:14:11.187342 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240501 14:14:11.192656 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:11.253288 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:14:11.257201 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:14:11.530031 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:11.530913 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:14:11.537317 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:14:11.589536 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:11.713460 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:11.773352 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:14:11.773519 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:14:11.779497 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:14:11.794684 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:14:11.801985 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:14:11.804949 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:14:11.828858 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:14:11.962888 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:14:12.003103 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:12.040724 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:14:12.084947 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:14:12.088155 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:12.119480 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:14:12.177306 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:12.186046 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:14:12.266654 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:14:12.274593 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:14:12.294950 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:14:12.363804 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:14:12.381016 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:14:12.459878 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:14:12.481554 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:14:12.482158 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:14:12.503684 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:14:12.546988 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:14:12.605582 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:12.606551 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:14:12.606979 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:12.636822 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:14:12.665967 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:12.742754 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:14:12.823374 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:14:12.840435 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:14:12.880899 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:14:12.932328 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:14:12.953604 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:14:13.002153 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:14:13.047040 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:14:13.148756 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:13.184190 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:13.194556 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:13.213892 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 14:14:13.226625 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:14:13.235314 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:14:13.325832 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:14:13.498368 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:14:13.555601 5322 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 145 similar messages]
W20240501 14:14:13.556697 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
I20240501 14:14:13.716650 17023 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:14:13.716841 17023 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:14:13.717748 17023 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:14:13.718152 5255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" is_pre_election: true
I20240501 14:14:13.718376 5255 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 2.
W20240501 14:14:13.719473 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:14:13.719655 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:14:13.719743 5321 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:14:13.720192 17023 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:14:13.742149 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:14:13.748198 5192 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 129 similar messages]
W20240501 14:14:13.749007 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240501 14:14:13.755000 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:13.766567 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:14:13.777626 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:13.959144 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:14:13.981606 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:14:14.030220 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:14:14.046494 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:14.049041 5062 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 137 similar messages]
W20240501 14:14:14.049789 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:14:14.095317 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:14:14.098353 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:14:14.190002 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:14:14.215862 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:14:14.236243 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:14.238262 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:14:14.259557 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:14:14.314190 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:14.383555 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:14:14.385154 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:14:14.410780 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:14:14.440328 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:14:14.483619 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:14.513686 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:14:14.559623 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:14:14.580901 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:14:14.640342 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:14.673276 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:14:14.677033 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:14:14.702180 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:14:14.719210 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:14.890836 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:14:14.939625 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:14.945667 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:14:14.981468 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:14:14.997804 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:14:15.032850 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:14:15.085347 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:15.147022 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:15.147354 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:14:15.158509 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:14:15.199486 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:14:15.202243 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:14:15.255764 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:14:15.358366 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:14:15.361508 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:15.441550 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:14:15.475804 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:14:15.498299 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:14:15.545728 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:15.549048 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:14:15.601390 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:15.656718 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:14:15.781646 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:15.793377 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:14:15.970896 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:14:15.987787 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:15.998148 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:14:16.019384 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:14:16.041850 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:16.071359 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:14:16.240593 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:16.275807 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:14:16.367620 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:16.389601 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:14:16.395598 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:14:16.401927 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:14:16.424479 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240501 14:14:16.444290 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:14:16.494613 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:14:16.537439 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:14:16.537474 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:14:16.554548 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:16.678411 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:16.722084 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:14:16.756354 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:14:16.771962 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:14:16.833076 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:14:16.852316 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:16.854750 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:14:16.856822 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:14:16.863426 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:14:16.947597 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:14:16.977046 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:14:17.004379 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:17.022603 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:14:17.060669 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:14:17.138865 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:14:17.145445 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:17.227764 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:14:17.255343 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:14:17.291777 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:14:17.430781 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:17.443859 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:14:17.479969 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:14:17.543367 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:14:17.546659 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:14:17.558876 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:14:17.586750 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:17.736372 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:14:17.763096 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:17.775606 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:14:17.812085 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:14:17.838194 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:14:17.914510 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:14:17.924437 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:17.925474 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:14:17.988273 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:18.005349 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:14:18.071342 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:14:18.216166 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 47 similar messages]
W20240501 14:14:18.248628 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:14:18.270238 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:14:18.349188 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:18.350632 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:14:18.431185 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:18.523797 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:14:18.558912 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:18.599689 5322 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 146 similar messages]
W20240501 14:14:18.661676 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:14:18.688650 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:14:18.725445 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:14:18.750741 5192 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 133 similar messages]
W20240501 14:14:18.777374 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:14:18.830309 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:14:18.854091 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:14:18.886178 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:14:18.920930 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:18.961463 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:14:18.971491 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:18.975450 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:14:19.004834 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:14:19.016925 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:19.060065 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 138 similar messages]
W20240501 14:14:19.060914 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240501 14:14:19.089610 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:14:19.137842 4838 catalog_manager.cc:5014] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet de6c7753c8564107b858876fcfa5924d with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER failed with leader 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }.
Pending config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } }
W20240501 14:14:19.191637 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:14:19.291214 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:14:19.308353 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:14:19.313977 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:14:19.352099 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:14:19.371686 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:19.371783 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:19.378254 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:14:19.404729 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:19.508492 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:14:19.513408 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:14:19.520849 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:14:19.553541 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:14:19.583703 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:14:19.593374 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:14:19.710251 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:14:19.769341 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:19.803738 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:14:19.905862 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:14:19.908005 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:19.954581 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:20.000300 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:14:20.043692 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:14:20.062157 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:14:20.105103 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:14:20.145195 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:14:20.203967 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:14:20.346557 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:20.379722 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:20.503293 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:14:20.508467 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:14:20.525174 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:20.550057 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:14:20.597923 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:14:20.655210 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:14:20.657919 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:14:20.811158 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:14:20.884681 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:20.925372 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:20.938719 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:20.966002 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:14:21.087986 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:14:21.142529 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:14:21.166465 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:14:21.177443 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:14:21.278620 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:14:21.353389 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:21.366346 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:14:21.382670 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:14:21.470026 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:21.472985 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:14:21.476686 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:14:21.491312 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240501 14:14:21.526688 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:14:21.530143 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:21.544941 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:14:21.570070 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:14:21.577178 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:14:21.591288 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
I20240501 14:14:21.622648 17523 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:14:21.622830 17523 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:14:21.623728 17523 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers be56773105e744919601127a94e98adf (127.29.124.132:35459), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:14:21.624233 5386 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf" is_pre_election: true
I20240501 14:14:21.624574 5386 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 2.
W20240501 14:14:21.625200 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:14:21.625401 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:14:21.625470 5191 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:14:21.625721 17523 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:14:21.696424 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:14:21.742408 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:14:21.748138 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:21.763051 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:14:21.775966 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:14:21.879036 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:14:21.890252 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:21.902228 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:14:22.037319 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:14:22.040985 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:14:22.117936 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:14:22.129307 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:14:22.139930 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:22.140211 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:22.268810 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:14:22.269981 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:14:22.303552 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:14:22.387660 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:14:22.434180 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:22.504868 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:14:22.511924 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:14:22.645826 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:22.645936 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:22.653684 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:14:22.724195 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:14:22.755129 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:14:22.759653 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:14:22.908231 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:14:22.951037 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:22.958387 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:14:22.986575 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:14:23.066092 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:23.066588 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:14:23.095506 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:14:23.165192 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:23.189960 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:14:23.295953 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:14:23.332357 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:14:23.442982 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 14:14:23.563411 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:23.566459 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:23.569072 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:14:23.574924 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:23.628329 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240501 14:14:23.629765 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:14:23.642946 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240501 14:14:23.710592 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:14:23.777423 5192 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 131 similar messages]
W20240501 14:14:23.838145 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:14:23.889482 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:14:23.894361 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:14:23.917152 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:14:23.936007 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:14:23.962806 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:23.979321 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:24.034690 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:14:24.065663 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:24.090822 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:14:24.104434 5062 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240501 14:14:24.171011 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:14:24.207034 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:14:24.207145 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:14:24.223719 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:14:24.263083 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:14:24.288852 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:14:24.297227 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:14:24.303133 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:14:24.311419 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:14:24.356973 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:14:24.414556 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:14:24.458298 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:24.492477 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:14:24.559834 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:24.562345 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:14:24.639097 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:14:24.643376 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:14:24.663801 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:24.670454 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:14:24.897593 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:14:24.938508 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:14:24.963333 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:24.999132 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:14:25.110976 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:25.128983 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:14:25.129992 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:25.202308 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:14:25.225490 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:14:25.245541 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:14:25.411001 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:14:25.424391 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:14:25.426560 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:14:25.466851 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:14:25.495687 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:25.505010 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:14:25.567261 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:25.638837 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:25.684974 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:14:25.887274 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:14:25.910182 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:14:25.919643 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:14:25.971771 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:26.058645 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:14:26.069940 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:14:26.110117 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:26.128525 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:26.155484 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:14:26.159150 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240501 14:14:26.215368 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:14:26.279346 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:14:26.375950 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:14:26.384171 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:14:26.405586 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:26.413583 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:14:26.464715 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:14:26.507618 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:14:26.534608 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:14:26.550160 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:14:26.598867 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:26.599006 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:26.621572 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:14:26.665688 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:14:26.689900 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:14:26.712575 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:14:26.844774 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:14:26.878125 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:26.882251 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:14:26.896009 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:14:26.899691 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:14:26.939005 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:14:26.956947 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:14:27.067719 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:27.085299 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:14:27.223913 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:27.253422 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:14:27.310496 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:14:27.339898 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:14:27.407636 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:27.448834 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:27.494863 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:14:27.517617 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:14:27.524369 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:14:27.556811 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:14:27.620328 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:14:27.665591 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:14:27.671425 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:27.740881 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:14:27.801069 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:14:27.832715 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:27.913673 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:14:27.916049 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:27.940078 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:14:27.961633 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:14:27.964309 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:14:28.160243 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:14:28.214759 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:14:28.233135 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:28.349978 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:14:28.407428 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:28.409581 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:14:28.470022 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 14:14:28.489596 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:14:28.527704 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:14:28.532968 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:28.631053 5322 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240501 14:14:28.701665 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:14:28.721721 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:14:28.753810 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:28.798419 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 129 similar messages]
W20240501 14:14:28.828497 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:14:28.889577 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240501 14:14:28.897652 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:14:28.918381 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:14:28.928956 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:14:28.951987 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:14:29.017655 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:29.019706 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:14:29.072081 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:14:29.089861 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:29.138525 5062 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240501 14:14:29.152132 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:14:29.154531 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:14:29.161072 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:14:29.172468 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:14:29.177531 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:14:29.263957 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:29.292220 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:14:29.333348 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:14:29.388231 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:14:29.425019 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:29.448009 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:14:29.452443 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:14:29.480312 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:14:29.595791 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:29.718284 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:14:29.733557 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:14:29.848743 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:29.875342 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:14:29.877326 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:29.889698 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:14:29.939330 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:14:29.988438 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:14:30.003482 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:14:30.003799 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:14:30.091181 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:30.100013 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:14:30.112721 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:14:30.124284 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:14:30.150892 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:14:30.233079 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:30.396078 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:30.467967 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:14:30.540961 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:30.567656 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:14:30.648413 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:30.656240 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:14:30.699368 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:14:30.764972 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:14:30.797998 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:14:30.803812 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:30.851599 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:14:30.878129 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:14:30.918422 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:14:30.957978 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:14:31.071929 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:31.118618 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:14:31.164300 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:31.237951 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:14:31.368381 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240501 14:14:31.400032 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:14:31.419215 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:31.448268 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:14:31.462684 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:14:31.527045 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:14:31.527918 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:14:31.534722 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:14:31.537215 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:31.570858 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:14:31.600603 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:14:31.600872 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:14:31.702896 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:14:31.706203 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:14:31.737906 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:31.772769 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:14:31.784507 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:14:31.786350 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:14:31.814004 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:14:31.832178 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:14:31.833163 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:31.911149 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:14:31.971982 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:14:31.998042 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:14:32.101338 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:14:32.103562 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:32.153841 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:32.201514 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:14:32.322818 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:14:32.398240 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:32.431668 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:14:32.443684 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:14:32.514878 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:14:32.555553 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:32.597121 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:32.610980 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:14:32.641623 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:14:32.646678 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:14:32.672181 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:14:32.695609 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:14:32.711712 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:14:32.774060 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:33.007251 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:14:33.057799 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:14:33.074720 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:33.082239 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:33.125183 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:14:33.167153 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:14:33.186417 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:14:33.210105 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:14:33.298308 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:33.483420 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:14:33.487533 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:14:33.507524 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 14:14:33.511644 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:14:33.559695 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:14:33.619127 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:33.684234 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:33.708931 5322 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 142 similar messages]
W20240501 14:14:33.737000 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
I20240501 14:14:33.782017 17921 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:14:33.782207 17921 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:14:33.783102 17921 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
W20240501 14:14:33.783921 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
I20240501 14:14:33.784133 5254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" is_pre_election: true
I20240501 14:14:33.784379 5254 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 2.
W20240501 14:14:33.784773 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:14:33.785543 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:14:33.785630 5321 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:14:33.785805 18170 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:14:33.786113 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:14:33.837594 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:14:33.843608 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:14:33.860666 5192 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 134 similar messages]
W20240501 14:14:33.861465 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:14:33.915059 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:14:34.119427 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:14:34.124181 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:14:34.138849 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240501 14:14:34.139369 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:14:34.139835 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240501 14:14:34.143962 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:14:34.174263 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:14:34.204406 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:14:34.223587 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:34.226449 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:14:34.269917 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:14:34.272330 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:14:34.274312 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:34.302521 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:14:34.325975 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:34.333868 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:14:34.459760 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:14:34.611335 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:34.679256 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:14:34.683344 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:14:34.688292 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:14:34.750792 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:34.756840 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:34.785530 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:14:34.867774 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:14:34.908509 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:14:34.913846 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:14:34.926961 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:14:34.962422 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:14:34.994499 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:14:35.047266 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:35.067956 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:14:35.158746 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:35.177022 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:14:35.274461 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:14:35.329368 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:14:35.330148 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:35.348044 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:14:35.359504 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:14:35.459789 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:14:35.543471 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:35.585808 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:14:35.588877 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:14:35.593523 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:14:35.619020 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:35.882323 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:35.885787 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:14:35.955108 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:14:36.039134 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:14:36.056780 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:14:36.079061 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:14:36.164698 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:36.217813 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:36.309314 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:14:36.311208 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:14:36.362329 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:14:36.393375 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:14:36.420728 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:36.439723 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:14:36.471757 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240501 14:14:36.507731 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:14:36.567214 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:14:36.575191 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:14:36.633019 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:14:36.635195 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:36.638907 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:14:36.676056 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:14:36.688613 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:14:36.730090 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:14:36.767618 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:36.820175 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:14:36.838241 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:14:36.852301 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:14:36.903661 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:36.988608 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:14:37.043821 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:14:37.123072 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:14:37.172390 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:14:37.178167 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:14:37.246495 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:14:37.259501 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:37.293216 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:37.314443 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:14:37.320879 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:37.359591 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:14:37.379223 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:14:37.408617 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:14:37.532390 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:14:37.564563 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:14:37.714826 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:14:37.746501 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:14:37.761369 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:14:37.853500 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:37.869210 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:37.895252 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:14:37.899448 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:37.916376 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:14:37.943291 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:14:38.135782 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:14:38.158895 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:14:38.257323 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:38.272994 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:14:38.295260 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:14:38.349766 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:38.404067 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:14:38.405189 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:38.481070 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:14:38.541716 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:14:38.541743 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:14:38.601619 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 14:14:38.615391 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240501 14:14:38.645296 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:38.678656 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:14:38.808079 5322 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 146 similar messages]
W20240501 14:14:38.808848 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:14:38.821734 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:14:38.825708 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:14:38.901309 5192 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 127 similar messages]
W20240501 14:14:38.901772 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:14:38.922899 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:14:38.941453 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:14:38.957564 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:38.996723 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:39.022356 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:39.071275 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:14:39.085999 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:14:39.127175 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:14:39.130609 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:14:39.144611 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:14:39.191375 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240501 14:14:39.213014 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:14:39.271813 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:14:39.284968 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:14:39.351091 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:39.366366 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:14:39.405750 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:14:39.552595 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:14:39.566617 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:39.593302 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:39.652195 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:14:39.710057 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:14:39.711526 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:14:39.771030 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:14:39.808291 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:14:39.859650 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:39.875898 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:14:39.985119 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:39.998665 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:14:40.051321 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:14:40.061194 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:14:40.095763 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:14:40.137063 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:40.214385 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:14:40.261231 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:14:40.268493 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:14:40.289611 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:14:40.424458 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:40.429659 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:14:40.436491 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
I20240501 14:14:40.463254 18504 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:14:40.463418 18504 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:14:40.464084 18504 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:14:40.464370 5255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" is_pre_election: true
I20240501 14:14:40.464587 5255 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 2.
W20240501 14:14:40.465348 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:14:40.465513 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:14:40.465602 5321 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:14:40.465842 18504 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:14:40.504667 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:14:40.579404 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:14:40.608275 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:40.774246 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:14:40.792093 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:14:40.860486 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:14:40.935338 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:40.937005 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:14:40.954721 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:40.998868 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:14:41.002673 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240501 14:14:41.006294 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:41.015556 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:14:41.120903 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:14:41.154141 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:14:41.260634 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:14:41.331545 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:14:41.410241 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:14:41.411823 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:14:41.447106 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:14:41.467165 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:14:41.469370 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:14:41.500658 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:14:41.507084 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:41.532820 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:41.555018 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:41.570837 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:14:41.713409 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:14:41.716073 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:14:41.800480 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:14:41.903956 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:14:41.942293 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:14:42.025620 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:14:42.026695 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:42.035205 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:14:42.050519 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:42.055635 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:42.130846 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:14:42.137224 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:14:42.161388 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:14:42.284713 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:14:42.328351 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:14:42.370332 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:14:42.380854 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:14:42.385427 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:14:42.392616 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:14:42.457348 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:14:42.518363 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:42.608974 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:42.660882 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:42.760247 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:14:42.798003 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:14:42.830557 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:14:42.890406 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:14:42.891047 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:14:42.896699 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:14:43.014273 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:43.130697 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:43.149564 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:14:43.162956 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:14:43.188805 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:14:43.227442 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:43.316831 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:14:43.388346 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:14:43.405975 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:14:43.459765 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:14:43.491990 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:43.493993 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:14:43.582962 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:14:43.612349 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:14:43.619812 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:43.640583 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240501 14:14:43.661233 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:14:43.673243 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:43.750093 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 14:14:43.773792 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:14:43.807963 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:14:43.880919 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240501 14:14:43.884796 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:14:43.896469 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:14:43.901151 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:14:43.926098 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:14:43.931528 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 130 similar messages]
W20240501 14:14:43.975564 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:14:43.992512 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:44.031466 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:14:44.063230 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:44.073331 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:44.183564 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:14:44.207098 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:14:44.315194 5062 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 145 similar messages]
W20240501 14:14:44.343851 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:14:44.348973 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:14:44.361567 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:14:44.421635 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:14:44.500468 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:44.536203 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:14:44.539125 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:14:44.541850 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:14:44.614445 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:44.632995 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:14:44.641620 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:44.687820 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:14:44.752657 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:14:44.822613 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:14:44.878715 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:14:44.914762 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:14:45.007285 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:45.090378 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:14:45.150174 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:45.204281 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:14:45.213550 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:45.267990 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:14:45.360862 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:14:45.389463 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:14:45.424690 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:14:45.482496 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:14:45.505806 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:45.540382 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:14:45.592859 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:14:45.643294 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:14:45.677384 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:14:45.736954 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:14:45.743067 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:45.821453 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:14:45.823320 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:45.860226 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:14:45.961913 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:14:45.962878 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:45.965936 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:14:45.988452 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:14:46.075572 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:14:46.143713 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240501 14:14:46.161347 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:14:46.200313 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:14:46.242442 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:14:46.257246 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:14:46.262174 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:14:46.331241 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:14:46.340164 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:46.377748 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:46.387307 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:14:46.470957 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:14:46.473212 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:14:46.515583 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:46.556612 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:14:46.643064 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:14:46.659623 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:14:46.835866 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:14:46.844902 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:14:46.891103 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:46.907447 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:14:46.931057 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:46.956979 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:14:46.974920 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:14:46.981017 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:14:46.987782 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:47.160845 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:14:47.192440 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:14:47.280401 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:14:47.317477 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:14:47.353334 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:47.367061 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:14:47.433861 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:14:47.444998 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:14:47.500154 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:47.511890 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:14:47.571895 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:47.613866 4840 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 13 similar messages]
W20240501 14:14:47.614858 4840 catalog_manager.cc:4573] TS 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 9d32e0707d244f2c9bbcb57f62b09f31: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
I20240501 14:14:47.651005 18769 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:14:47.651161 18769 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:14:47.651834 18769 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:14:47.652168 5255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" is_pre_election: true
I20240501 14:14:47.652403 5255 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 2.
W20240501 14:14:47.653517 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:14:47.653692 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:14:47.653769 5321 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:14:47.653983 18769 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:14:47.730191 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:14:47.751188 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:14:47.788177 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:14:47.876971 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:14:47.909746 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:47.929131 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:14:47.956141 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:14:47.968186 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:14:48.084693 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:48.108528 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:14:48.110945 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:14:48.126724 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:14:48.148069 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:48.316083 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:14:48.343976 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:14:48.355631 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:48.398586 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:14:48.472705 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240501 14:14:48.532681 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:48.554157 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:14:48.596226 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:14:48.670171 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:14:48.683171 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:48.727859 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:14:48.729704 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:14:48.812652 4840 catalog_manager.cc:4573] TS 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 61a274b24f53458f88c2cde4f8dbe4f4: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:14:48.819303 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:14:48.839515 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:14:48.859777 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:14:48.894026 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240501 14:14:48.894778 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:14:48.921181 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:14:48.930004 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 53 similar messages]
W20240501 14:14:48.942919 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:14:48.946507 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 128 similar messages]
W20240501 14:14:48.980777 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:48.982416 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:14:49.130204 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:14:49.134701 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:49.139484 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:14:49.159610 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:14:49.196831 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:14:49.204108 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:49.270220 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:14:49.324419 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:14:49.353055 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 143 similar messages]
W20240501 14:14:49.353823 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:14:49.430595 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:14:49.472870 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:49.514209 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:49.528295 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:14:49.580617 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:49.623618 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:14:49.789508 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:14:49.860374 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:14:49.942730 4840 catalog_manager.cc:4573] TS 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet fee16905387d428aa02cd2070c86c790: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:14:49.949543 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:14:49.980108 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:14:49.983244 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:14:50.017843 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:14:50.039391 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:14:50.073310 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:50.120275 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:50.150624 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:50.206473 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240501 14:14:50.296779 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:14:50.306330 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:14:50.359977 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:14:50.416507 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:14:50.430137 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:14:50.446380 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:14:50.495922 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:14:50.646178 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:50.657819 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:14:50.728863 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:50.739024 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:50.757550 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:14:50.774458 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:14:50.813306 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:14:50.885774 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
I20240501 14:14:50.887918 19035 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:14:50.888077 19035 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:14:50.888914 19035 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers be56773105e744919601127a94e98adf (127.29.124.132:35459), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:14:50.889482 5384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf" is_pre_election: true
I20240501 14:14:50.889710 5384 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 2.
W20240501 14:14:50.891461 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:14:50.891620 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:14:50.891705 5191 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:14:50.891921 19035 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:14:50.942442 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:14:51.046326 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:14:51.079142 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:14:51.088941 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:14:51.128175 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:51.130707 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:51.169906 4840 catalog_manager.cc:4573] TS 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet c8a961e711114f369ba06840c062b982: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
W20240501 14:14:51.237133 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:14:51.244902 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:14:51.247447 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:14:51.304109 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:51.367202 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:14:51.381678 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:14:51.388150 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:14:51.453325 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:14:51.463877 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:14:51.477854 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:14:51.508383 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:14:51.525480 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:51.633054 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:14:51.634213 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:14:51.750015 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:51.772693 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:14:51.829789 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:14:51.884352 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:51.910658 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:14:51.978974 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:51.982805 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:14:52.042950 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:14:52.064970 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:14:52.103360 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:14:52.317584 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:14:52.318410 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:52.352699 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:14:52.414129 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:52.427944 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:14:52.489993 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:52.557324 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:14:52.558626 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:14:52.580582 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:14:52.647343 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:14:52.721218 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:14:52.725936 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:14:52.787576 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:52.820183 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:14:52.845845 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:14:52.882882 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:52.903486 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:14:52.926266 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:14:52.941531 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:52.972466 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:14:53.192097 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:14:53.206118 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:14:53.236606 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:53.270025 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:53.303669 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240501 14:14:53.374431 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:14:53.404927 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:53.420027 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:14:53.468732 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:14:53.486718 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:14:53.535272 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:14:53.547873 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:14:53.665427 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:53.686712 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:14:53.759297 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:53.759078 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:14:53.760233 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:14:53.787895 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:14:53.798295 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:14:53.849969 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:14:53.857172 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:14:53.871031 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:14:53.933580 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 14:14:53.943305 5322 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240501 14:14:53.972344 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:14:54.005285 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:14:54.014523 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 133 similar messages]
W20240501 14:14:54.015343 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:54.105633 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:54.122982 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:14:54.180217 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:14:54.197014 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:54.267766 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:14:54.286324 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:14:54.317550 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:14:54.353864 5062 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 136 similar messages]
W20240501 14:14:54.403662 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:14:54.422122 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:14:54.475708 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:14:54.526041 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:14:54.565659 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:54.580224 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:54.586975 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:14:54.657374 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:14:54.737637 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:54.899462 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:14:54.911263 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:14:55.007773 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:14:55.011116 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:14:55.119189 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:55.121619 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:55.150736 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:14:55.152035 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:14:55.195887 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:14:55.220083 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:14:55.230136 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:14:55.266175 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:14:55.312216 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:55.376372 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:14:55.542240 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:14:55.549885 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:14:55.574589 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:55.698783 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:55.721035 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:55.778519 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240501 14:14:55.807442 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:14:55.888322 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:14:55.931946 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:14:55.985461 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:14:55.991214 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:14:56.026504 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:14:56.060705 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:56.082054 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:56.106956 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:56.115440 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:14:56.162954 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:14:56.187678 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:14:56.219192 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:14:56.245308 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:14:56.292789 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:14:56.297994 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:14:56.402681 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:14:56.414467 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:14:56.435282 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:14:56.467540 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:14:56.641875 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:56.649897 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:56.715847 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:14:56.722327 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:14:56.723555 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:14:56.728226 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:56.789852 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:14:56.798239 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:14:56.851720 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:14:56.863369 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:14:56.885299 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:14:57.010159 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:14:57.030535 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:14:57.032768 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:14:57.113816 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:57.130229 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:57.134918 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:14:57.169072 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:57.353016 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:14:57.459852 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:14:57.468390 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:14:57.499701 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:14:57.549914 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:57.589998 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:14:57.601878 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:14:57.632483 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:57.641721 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:14:57.743218 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:14:57.754076 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:57.830123 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:14:57.872627 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:14:57.926947 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:14:58.045611 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:14:58.063208 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:58.139829 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:14:58.210950 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:58.221688 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:14:58.252559 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:14:58.271796 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:58.413486 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:14:58.428247 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240501 14:14:58.474761 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:14:58.541138 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:14:58.626626 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:14:58.641209 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:58.646659 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:14:58.655697 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:14:58.660135 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:14:58.660559 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:14:58.683707 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:58.818354 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:14:58.820079 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:14:58.860328 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
I20240501 14:14:58.949685 19462 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:14:58.949837 19462 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:14:58.950613 19462 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers be56773105e744919601127a94e98adf (127.29.124.132:35459), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
W20240501 14:14:58.950881 5322 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 138 similar messages]
I20240501 14:14:58.951010 5384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf" is_pre_election: true
I20240501 14:14:58.951198 5384 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 2.
W20240501 14:14:58.952795 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:14:58.952970 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:14:58.953049 5191 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:14:58.953279 19462 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:14:58.956643 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:14:58.964593 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:14:58.982093 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:14:59.026499 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 133 similar messages]
W20240501 14:14:59.027241 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:59.076762 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:14:59.082811 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 14:14:59.083786 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:14:59.141806 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:14:59.146275 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:59.185483 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:14:59.204963 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:14:59.216032 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:14:59.238317 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:14:59.287257 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:14:59.291311 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:59.350392 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:14:59.394510 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:14:59.407820 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240501 14:14:59.487352 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:14:59.594933 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:14:59.622949 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:14:59.637908 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:14:59.644805 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:14:59.648003 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:59.684228 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:14:59.743793 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:14:59.823645 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:14:59.934626 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:14:59.967626 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:14:59.982820 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:15:00.024211 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:15:00.060014 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:15:00.110235 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:15:00.110373 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:00.154464 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:00.248335 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:15:00.255244 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:15:00.381678 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:15:00.457800 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:15:00.463019 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:15:00.513573 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:15:00.530982 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:15:00.544469 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:00.720976 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:00.786738 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:15:00.880738 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:15:00.886050 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240501 14:15:00.933497 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:15:00.963209 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:15:00.993882 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:15:01.061834 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:15:01.064312 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:15:01.117393 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:15:01.127967 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:15:01.139321 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:15:01.159389 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:01.183238 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:15:01.234505 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:15:01.258186 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:01.307240 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:15:01.393446 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:15:01.437575 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:15:01.448451 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:15:01.526988 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:15:01.536132 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:15:01.560662 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:15:01.614980 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:01.634223 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:15:01.680730 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:15:01.694509 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:15:01.755879 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:01.774495 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:15:01.777076 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:15:01.788365 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:15:01.853417 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:15:01.862478 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:15:01.871001 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:15:01.951071 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:15:01.967489 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:15:01.968948 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:15:02.079015 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:15:02.092797 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:02.183423 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:15:02.353338 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:15:02.365718 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:02.415568 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:15:02.488011 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:15:02.513377 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:15:02.579244 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:02.654901 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:15:02.655251 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:15:02.668457 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:15:02.709794 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:15:02.758982 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:15:02.811735 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:15:02.867694 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:15:02.937381 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:15:02.978364 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:03.023157 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:03.073724 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:15:03.083909 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:15:03.257191 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:15:03.267271 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:15:03.306365 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:15:03.383612 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:15:03.412528 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240501 14:15:03.433773 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:15:03.438129 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:03.438601 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:15:03.524024 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:15:03.537603 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:15:03.571475 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:15:03.623214 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:03.640275 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:15:03.684479 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:15:03.710844 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:15:03.725553 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:15:03.788355 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:15:03.854975 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:03.936297 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:15:03.975641 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 140 similar messages]
W20240501 14:15:03.987645 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:15:03.994145 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:15:04.033519 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 129 similar messages]
W20240501 14:15:04.059168 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:15:04.063410 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:15:04.085041 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:15:04.088488 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:04.099647 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:15:04.110064 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:15:04.116029 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 14:15:04.220907 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:15:04.228076 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:15:04.252350 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:15:04.399956 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:15:04.410687 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:15:04.436228 5062 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240501 14:15:04.437124 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:15:04.454169 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:15:04.454166 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:04.470654 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:15:04.472743 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:04.493892 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:15:04.578472 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:15:04.610149 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:15:04.644976 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:15:04.844292 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:15:04.905737 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:15:04.916697 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:04.942410 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:04.987706 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:15:05.017414 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:15:05.158860 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:15:05.283901 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:15:05.294697 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:15:05.327237 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:15:05.350535 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:15:05.420287 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:05.458549 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:15:05.464812 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:05.528087 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:15:05.562022 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:15:05.614596 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:15:05.622421 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:15:05.627390 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:15:05.653918 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:15:05.782490 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:15:05.786681 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:15:05.820639 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:05.872002 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:15:05.910940 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:15:05.978319 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:15:05.992733 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:15:06.005290 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:06.049592 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240501 14:15:06.091296 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:15:06.145730 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:15:06.150583 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:15:06.210594 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:15:06.266980 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:06.291217 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:15:06.324707 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:15:06.347815 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:15:06.491022 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:15:06.495726 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:15:06.525115 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:15:06.538043 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:06.551596 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:15:06.568869 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:15:06.592636 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:15:06.595695 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:15:06.615217 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:15:06.704329 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:15:06.824827 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:15:06.833920 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:06.847616 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:15:06.860605 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:15:06.915941 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:15:06.939811 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:15:06.962882 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:15:06.990960 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:15:06.999014 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:15:07.068724 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:07.070111 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:15:07.249302 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:07.300591 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:15:07.323127 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:15:07.412119 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:15:07.461244 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:07.527283 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:15:07.578217 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:15:07.619967 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:15:07.717377 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:15:07.733076 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:15:07.738847 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:07.791215 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:15:07.844722 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:15:07.913278 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:15:08.026248 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:08.027709 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:15:08.040499 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:15:08.129145 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:08.132671 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:15:08.146442 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:15:08.220849 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:15:08.321805 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:15:08.330773 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:15:08.368029 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:15:08.415194 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:15:08.551147 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:08.564131 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:15:08.591765 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240501 14:15:08.602064 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:15:08.686975 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:08.731050 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:15:08.737071 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:15:08.770671 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:15:08.770861 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:15:08.779400 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:15:08.795612 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:15:08.830418 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:15:08.898681 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:15:08.916504 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:15:08.917554 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:15:09.013999 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240501 14:15:09.014951 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:15:09.105891 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:15:09.109443 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 134 similar messages]
W20240501 14:15:09.117142 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:09.117410 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:15:09.125636 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:15:09.146529 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240501 14:15:09.147665 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:15:09.163874 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:15:09.179033 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:15:09.203758 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:15:09.208834 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:15:09.300405 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:15:09.300555 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:09.318573 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:15:09.320518 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:15:09.325644 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:15:09.421746 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
I20240501 14:15:09.469424 19937 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:15:09.469610 19937 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:15:09.470484 19937 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:15:09.471024 5254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" is_pre_election: true
I20240501 14:15:09.471253 5254 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 2.
W20240501 14:15:09.472725 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:15:09.472893 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:15:09.472967 5321 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:15:09.473364 19937 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:15:09.485646 5062 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 143 similar messages]
W20240501 14:15:09.498797 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:09.585942 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:15:09.672641 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:15:09.711140 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:15:09.868990 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:15:09.873065 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:09.963590 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:09.969415 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:15:10.077127 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:15:10.077153 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:15:10.151850 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:15:10.197364 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:15:10.230998 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:15:10.282060 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:15:10.396235 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:10.480937 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:15:10.497361 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:15:10.505599 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:10.594411 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:15:10.599010 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:15:10.661082 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:15:10.735977 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:15:10.739732 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:15:10.785166 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:10.810621 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:15:10.906508 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:15:10.988474 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:15:11.014693 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:15:11.024933 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:11.163730 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:15:11.185914 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:15:11.202564 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:15:11.208967 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:15:11.217232 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:15:11.243952 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240501 14:15:11.244199 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:11.258133 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:15:11.290324 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:15:11.314172 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:15:11.330837 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:15:11.362640 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:15:11.522305 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:11.545603 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:15:11.571584 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:15:11.572278 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:15:11.574796 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:15:11.700459 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:11.715611 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:15:11.779883 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:15:11.850062 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:15:11.895381 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:15:11.911700 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:15:11.915591 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:15:11.925001 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:15:11.925199 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:15:11.978642 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:15:11.984797 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:15:12.031598 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:15:12.043443 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:15:12.116173 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:12.299268 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:12.317849 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:15:12.373279 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:15:12.378271 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:15:12.563052 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:15:12.611200 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:12.624358 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:15:12.763617 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:15:12.787170 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:15:12.821942 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:15:12.822811 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:12.907989 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:15:12.913666 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:15:13.075402 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:15:13.089332 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:15:13.164304 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:15:13.188863 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:15:13.197698 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:13.236128 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:15:13.267506 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:15:13.300701 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:13.301522 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:15:13.319978 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:15:13.347404 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:15:13.385157 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:15:13.429235 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:15:13.553396 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:15:13.625766 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:15:13.639811 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:15:13.640563 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:13.651219 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:15:13.681684 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:15:13.744177 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240501 14:15:13.766958 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:15:13.806941 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:15:13.837710 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:15:13.860276 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:13.867579 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:15:13.892122 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:15:13.979581 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:15:13.993438 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:15:14.048249 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 144 similar messages]
W20240501 14:15:14.119055 5192 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 127 similar messages]
W20240501 14:15:14.120183 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:15:14.150252 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:15:14.151345 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:15:14.152673 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:15:14.232699 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:15:14.243978 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:14.270479 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:15:14.329968 4930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240501 14:15:14.350262 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:15:14.352571 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:15:14.367144 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:14.402314 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:15:14.452404 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:15:14.455173 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:15:14.472649 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:15:14.513829 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 137 similar messages]
W20240501 14:15:14.514770 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:15:14.602572 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:15:14.661688 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:15:14.681529 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:14.906603 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:15:14.945398 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:15:14.956454 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:14.987682 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:15:15.034883 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:15:15.090595 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:15:15.135324 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:15.149966 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:15:15.273898 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:15:15.304270 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:15:15.459812 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:15.497380 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:15:15.558118 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:15:15.559700 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:15:15.694757 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:15.696429 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:15:15.743168 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:15:15.748473 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:15:15.751057 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:15:15.758400 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:15:15.772866 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:15:15.800697 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:15:15.874500 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:15.961833 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:15:15.974956 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:15:16.030033 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:15:16.076191 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:16.076953 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:15:16.083914 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240501 14:15:16.100548 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:15:16.230038 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:15:16.265010 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:15:16.292227 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:15:16.375231 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:15:16.380342 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:15:16.445819 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:15:16.469516 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:15:16.488739 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:15:16.497753 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:16.498381 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:15:16.501958 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:15:16.519109 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:15:16.536250 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:15:16.549964 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:16.585378 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:15:16.640336 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:15:16.677079 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:15:16.719645 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:15:16.751262 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:15:16.793623 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:15:16.811818 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:15:16.950778 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:15:16.981673 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:15:16.983371 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:15:17.025403 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:15:17.033730 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:17.056774 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:15:17.070403 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:17.181574 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:15:17.381526 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:15:17.395771 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:15:17.505887 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:15:17.551601 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:15:17.567869 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:15:17.583876 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:17.652392 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:17.683341 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:15:17.775276 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:15:17.833636 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:15:17.886827 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:15:18.080921 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:15:18.086120 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:15:18.203089 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:18.233148 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:15:18.263764 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:15:18.272145 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:18.299943 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:15:18.334635 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:15:18.357344 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:15:18.387363 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:15:18.447422 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:15:18.522861 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:15:18.523553 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:15:18.530686 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:15:18.616402 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:15:18.620577 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 14:15:18.633215 20253 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:15:18.633459 20253 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:15:18.634259 20253 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:15:18.634968 5254 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "be56773105e744919601127a94e98adf" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" is_pre_election: true
I20240501 14:15:18.635208 5254 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate be56773105e744919601127a94e98adf in term 2.
W20240501 14:15:18.636041 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:15:18.636198 5321 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:15:18.636276 5321 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:15:18.636513 20253 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:15:18.649765 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:15:18.649884 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:18.655126 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:15:18.724674 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:15:18.746331 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240501 14:15:18.765687 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:18.851928 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:15:18.873703 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:15:18.912043 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:15:18.930783 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:15:18.938829 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:15:18.991101 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:15:19.001158 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:15:19.010828 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:15:19.060461 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 141 similar messages]
W20240501 14:15:19.064208 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:15:19.093489 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:19.100955 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:15:19.131785 5192 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 131 similar messages]
W20240501 14:15:19.142442 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:15:19.151684 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:15:19.162809 4838 catalog_manager.cc:5014] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet de6c7753c8564107b858876fcfa5924d with cas_config_opid_index 9: ChangeConfig:ADD_PEER:NON_VOTER failed with leader 1c3fdc5272484ac09ef3e5e36a08f27c (127.29.124.131:33209) due to error UNKNOWN_ERROR; will retry: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }.
Pending config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } } peers { permanent_uuid: "6dac90f01e744f9c8701a404440a4c77" member_type: VOTER last_known_addr { host: "127.29.124.136" port: 43377 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } } peers { permanent_uuid: "19a6d2d8a08142d5bffcde3a37234692" member_type: NON_VOTER last_known_addr { host: "127.29.124.134" port: 40665 } attrs { promote: true } }
W20240501 14:15:19.183382 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:15:19.198462 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:15:19.234848 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:15:19.300864 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:19.327860 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:15:19.371169 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:15:19.412145 4931 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240501 14:15:19.473826 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:15:19.505311 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:19.554055 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:15:19.562801 5062 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 146 similar messages]
W20240501 14:15:19.642270 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:15:19.648782 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:15:19.688863 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:15:19.874364 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:15:19.904513 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:19.997625 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:20.032330 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:15:20.118228 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:15:20.134163 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:15:20.142740 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:15:20.212365 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:15:20.290541 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:20.291961 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:15:20.445119 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:20.472661 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:15:20.520948 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:15:20.615505 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:15:20.699558 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:15:20.836192 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:15:20.849493 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:15:20.876812 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:20.891777 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:15:20.911252 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:15:20.930480 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:15:20.976066 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
I20240501 14:15:20.985965 20384 raft_consensus.cc:482] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240501 14:15:20.986111 20384 raft_consensus.cc:504] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "be56773105e744919601127a94e98adf" member_type: VOTER last_known_addr { host: "127.29.124.132" port: 35459 } } peers { permanent_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" member_type: VOTER last_known_addr { host: "127.29.124.131" port: 33209 } attrs { replace: true } } peers { permanent_uuid: "4360b59e3a154688a79ca5f9c0a37bc9" member_type: VOTER last_known_addr { host: "127.29.124.135" port: 35329 } attrs { promote: false } } peers { permanent_uuid: "938a24b0e6a040b7966225af26558ff8" member_type: VOTER last_known_addr { host: "127.29.124.133" port: 35635 } attrs { promote: false } }
I20240501 14:15:20.986974 20384 leader_election.cc:288] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers be56773105e744919601127a94e98adf (127.29.124.132:35459), 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329), 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635)
I20240501 14:15:20.987246 5384 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "51cbcf4888c440c2ad096c988ab2d968" candidate_uuid: "1c3fdc5272484ac09ef3e5e36a08f27c" candidate_term: 3 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: false dest_uuid: "be56773105e744919601127a94e98adf" is_pre_election: true
I20240501 14:15:20.987473 5384 raft_consensus.cc:2454] T 51cbcf4888c440c2ad096c988ab2d968 P be56773105e744919601127a94e98adf [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1c3fdc5272484ac09ef3e5e36a08f27c in term 2.
W20240501 14:15:20.988232 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111)
W20240501 14:15:20.988456 5191 leader_election.cc:334] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111)
I20240501 14:15:20.988596 5191 leader_election.cc:302] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 4 responses out of 4 voters: 2 yes votes; 2 no votes. yes voters: 1c3fdc5272484ac09ef3e5e36a08f27c, be56773105e744919601127a94e98adf; no voters: 4360b59e3a154688a79ca5f9c0a37bc9, 938a24b0e6a040b7966225af26558ff8
I20240501 14:15:20.988802 20384 raft_consensus.cc:2726] T 51cbcf4888c440c2ad096c988ab2d968 P 1c3fdc5272484ac09ef3e5e36a08f27c [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20240501 14:15:21.002727 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:15:21.024171 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:15:21.038832 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:15:21.061018 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:21.073559 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:15:21.101411 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:15:21.106882 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:15:21.151823 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:15:21.196045 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240501 14:15:21.230870 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:15:21.303443 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:21.316269 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:15:21.397821 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:15:21.424311 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:15:21.430212 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:15:21.433125 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:15:21.435829 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:15:21.443418 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:15:21.503430 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:15:21.551468 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:15:21.555450 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:15:21.593761 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:15:21.668692 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:15:21.673115 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:21.691679 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:15:21.830413 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:21.858641 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:15:21.867147 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:15:21.882653 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:15:21.940986 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:15:22.042074 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:15:22.053324 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:15:22.118431 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:15:22.150543 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:15:22.202872 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:15:22.232270 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:22.265537 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:22.382592 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:15:22.448817 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:15:22.507927 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:15:22.531064 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:15:22.653493 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:22.719066 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:15:22.732627 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:15:22.807474 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:15:22.827049 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:15:22.829221 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:22.878715 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:15:22.909772 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:15:23.074239 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:15:23.271303 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:23.322064 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:15:23.322470 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:15:23.328078 5321 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:15:23.339758 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:23.414054 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:15:23.414692 5061 consensus_peers.cc:485] T a726769d24284fedadd8e9e63cd27d05 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240501 14:15:23.456724 5061 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:15:23.457974 5322 consensus_peers.cc:485] T b284aa82594a44f5885f32fa05844913 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:15:23.489719 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:15:23.490989 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:15:23.508024 5062 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:15:23.530859 5322 consensus_peers.cc:485] T 2c39b53867e047faaf24cc6f87a8100c P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:15:23.548216 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:15:23.632789 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:15:23.662480 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:23.695072 5191 consensus_peers.cc:485] T 404e2851055541f0be1bb0971a7cc72a P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:15:23.709300 5192 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:15:23.718062 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:23.820176 5062 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:15:23.843971 5062 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:15:23.851425 5192 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:15:23.855717 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:15:24.003839 5322 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:15:24.041894 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:24.048815 5321 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:15:24.066366 5192 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:15:24.078873 5321 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240501 14:15:24.154184 5321 consensus_peers.cc:485] T f4d2b8c005b64bbb8e6621da068d15e5 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:15:24.208500 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Error code: TABLET_NOT_FOUND (6). Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:15:24.216145 5061 consensus_peers.cc:485] T 43b46fcf827c4275a47291afb107c290 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:15:24.221490 5191 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111) [suppressed 132 similar messages]
W20240501 14:15:24.237941 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:15:24.254563 4930 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:15:24.262609 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:24.280947 5191 consensus_peers.cc:485] T 23ca2bf1e59f44fcac8a48aff8d056b1 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:15:24.296571 5322 consensus_peers.cc:485] T e7d5f02ccba34254a4d20951878b1214 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:15:24.304649 5321 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:15:24.312639 5061 consensus_peers.cc:485] T ac4b234f794149e4b6def7a3b358942d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:15:24.349188 5321 consensus_peers.cc:485] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:15:24.466172 4931 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240501 14:15:24.468056 5191 consensus_peers.cc:485] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:15:24.586797 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:24.615732 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:15:24.616115 5061 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111) [suppressed 139 similar messages]
W20240501 14:15:24.616309 5191 consensus_peers.cc:485] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:15:24.621280 5061 consensus_peers.cc:485] T 3d6859d53fd747a5a5e94722626fa522 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 938a24b0e6a040b7966225af26558ff8 (127.29.124.133:35635): Couldn't send request to peer 938a24b0e6a040b7966225af26558ff8. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.133:35635: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:15:24.637887 5192 consensus_peers.cc:485] T 420642af520d45ffbc7f2d35f93e75a7 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:15:24.669644 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:24.866111 5062 consensus_peers.cc:485] T 9afe3b20a43d4b5bb800af39f945948d P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:15:24.992197 5191 consensus_peers.cc:485] T de6c7753c8564107b858876fcfa5924d P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:15:25.002224 5191 consensus_peers.cc:485] T 1a7bbfc15a76450bbc9197037e856e08 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:15:25.026587 5062 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:15:25.033668 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:15:25.039264 5062 consensus_peers.cc:485] T fee16905387d428aa02cd2070c86c790 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:15:25.050634 5062 consensus_peers.cc:485] T 9d32e0707d244f2c9bbcb57f62b09f31 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240501 14:15:25.065083 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:25.221148 5322 consensus_peers.cc:462] T 71bdb382b08341218c8bf2459fe65fc0 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111)
W20240501 14:15:25.408215 5322 consensus_peers.cc:485] T 9b88c43cb8704583a3e65c5c35c205f6 P be56773105e744919601127a94e98adf -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:15:25.529392 5321 consensus_peers.cc:485] T 61a274b24f53458f88c2cde4f8dbe4f4 P be56773105e744919601127a94e98adf -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:15:25.579335 5061 consensus_peers.cc:485] T f9cb784f60774ea7b6e70ccfebf867a8 P e9c60449cf3c46f8aca4ab6c535302a4 -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Couldn't send request to peer 19a6d2d8a08142d5bffcde3a37234692. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240501 14:15:25.595779 5191 consensus_peers.cc:462] T ec1e81011c6a46bc8a39aecf3a5e71e4 P 1c3fdc5272484ac09ef3e5e36a08f27c -> Peer 19a6d2d8a08142d5bffcde3a37234692 (127.29.124.134:40665): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.29.124.134:40665: connect: Connection refused (error 111)
W20240501 14:15:25.632236 4931 consensus_peers.cc:485] T 8cc5eec4600640f7a293877c825f8d96 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 6dac90f01e744f9c8701a404440a4c77 (127.29.124.136:43377): Couldn't send request to peer 6dac90f01e744f9c8701a404440a4c77. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.136:43377: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:15:25.683873 4930 consensus_peers.cc:485] T ba6cba753e934e719c730593c0706346 P 04088ee3bea440a7bd5ff772152b9410 -> Peer 4360b59e3a154688a79ca5f9c0a37bc9 (127.29.124.135:35329): Couldn't send request to peer 4360b59e3a154688a79ca5f9c0a37bc9. Status: Network error: Client connection negotiation failed: client connection to 127.29.124.135:35329: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240501 14:15:25.688444 5192 consensus_peers.cc:462] T 9fd94635797d447cbdc9181ee91f465b P 1c3fdc5272484ac09ef3e5e36a08f27c ->